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ABSTRACT 


Command, Control, Communications, Computers, and Intelligence (C4I) systems, 
each originally designed to address a single warfighting function, have been assembled 
into an interdependent C4I System of Systems (SoS). The C4I SoS continues to evolve, 
without overarching capabilities-based performance requirements. Without 
requirements, there is no practical, repeatable, and objective process to assess changes to 
the SoS. This project applied a disciplined systems engineering process to design a Joint 
C4I Capability Certification Measures (JC3M) system. JC3M can be used to define 
performance measures for a C4I SoS, determine baseline SoS performance, assess 
proposed SoS changes, and monitor SoS performance. Modeling and simulation tools 
were used to project the performance of three existing alternatives and two new 
alternatives. A Life Cycle Cost Estimate (LCCE) was generated for each alternative. An 
Analysis of Alternatives compared performance and cost. The Joint Test and Evaluation 
Methodology Capability Test Methodology (JTEM CTM) was projected to provide 
slightly better performance than other alternatives, at the median LCCE. The results 
were a recommendation to monitor JTEM CTM as it completes development, and 
employ the JTEM CTM in a C4I SoS evaluation to confirm its estimated cost and 
performance. 
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EXECUTIVE SUMMARY 

Historically, Command, Control, Communications, Computers, and Intelligence 
(C4I) systems were each developed to address a single warfighting function, whether 
used in a single Service or as a Joint solution. C4I systems, and their use, have changed 
in response to advances in technology as well as the growing complexity, scope, and 
tempo of warfighting. Individual C4I systems, whether changed or unmodified, have 
been assembled by developers and the operating forces into an interdependent System of 
Systems (SoS) in order to achieve effects which a single system could not achieve. 

The C4I SoS has grown, evolving in an almost biological manner without benefit 
of an overarching architecture, engineering design, or performance requirements. 

Without a SoS-level architecture, the effects of system-level changes on performance, 
capabilities, or standards compliance of the SoS are impossible to assess in an objective, 
repeatable, measurable, and effective manner. 

Individual systems are designed, developed, tested, and fielded in response to 
system-level requirements. Individual systems undergo interoperability certification, 
which ensures compliance to standards, but does not measure SoS effectiveness. Without 
measures of capabilities-based performance for the C4I SoS, system-level changes have 
an unknown effect on the C4I SoS; the operating forces can be saddled with systems that 
do not work as assembled. 

For example, when a Forward Observer (FO) sends a “Call For Fire,” position 
location information, fire control systems, and communications systems are used to 
request artillery, rockets, or naval gunfire effects on a target. There is extensive doctrine 
for the conduct of fire support, but there are not documented testable values which can be 
used to assess the success of a Call for Fire or the C4I systems supporting the task. If a 
FO had to initiate a Call For Fire five times, did the C4I SoS demonstrate a successful 
capability to provide fire support? What if the FO had to request seven times? If a 
change to a component system required the FO to initiate eight times, rather than seven, 
is the SoS still effective? Without performance measures for the C4I SoS, accurate 
fielding decisions for SoS components cannot be made. 
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The lack of performance measures for the C4I SoS is a well-known issue, but the 
C4I SoS will not be completely rebuilt in order to include performance measures; there 
are too many legacy systems, with significant inertia, that will continue to be used. 
Architecting the C4I SoS as a coherent whole (design from the top down) represents an 
improbable shift in philosophy. The “system-centric” (bottom up) approach is likely to 
continue. 

The Joint Capabilities Integration and Development System (JCIDS) is in place to 
ensure new systems are developed in response to required capabilities [CJCSI 3170.0IF, 
2007:2], and that those systems are "born joint." However, urgent unfunded needs 
statements, urgent warfighter needs, and the continued "biological growth" of the SoS 
make the need for a system to define performance measures an enduring one. 

The purpose of this project was to develop the Joint C4I Capability Certification 
Measures (JC3M) system, which could define threshold performance values for the C4I 
SoS. A disciplined and iterative systems engineering process was applied throughout the 
project. Stakeholders were identified at representative C4I test organizations, Subject 
Matter Experts (SMEs) were consulted, and their responses were used to elicit and 
validate quantifiable requirements. A functional hierarchy along with non-functional 
requirements, complete with evaluation measures to compare the performance of 
alternatives, was created for JC3M, and validated through SME consultation. 

Alternatives to be considered in the systems engineering process consisted of both 
existing systems and new conceptual systems. A review of the literature, combined with 
SME consultation, identified three existing systems that were potential JC3M alternative 
solutions. Two projects have been established recently to address the lack of 
performance measures for the C4I SoS. They are the Joint Test and Evaluation 
Methodology (JTEM) project’s Capability Test Methodology (CTM) under the Director 
of Operational Test and Evaluation and the Marine Air Ground Task Force (MAGTF) 

C4I Capability Certification Testing (MC3T) project within Marine Corps Systems 
Command. A third existing system, the Federation Of Systems (FEDOS) process is used 
by MCTSSA. Two new and unique solutions were created in order to add breadth to the 
comparisons. 
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During the design of the new alternatives, it was discovered that the initial goal of 
creating a system to define threshold performance values for the C4I SoS was not viable. 
The potential number of configurations of the C4I SoS, combined with the number of 
conditions in which it could be operated, created a practically infinite number of test 
conditions. Further, DoD doctrine [CJCSM 3500.04D, 2005, A-3] stipulated that local 
commanders, in response to the varied conditions their units operated in, could dictate 
their own performance thresholds. Any threshold performance values defined by a test 
organization might not be considered as representative of user needs, and thus might not 
be useful. These discoveries caused the reevaluation of requirements, and an associated 
redesign of the new alternatives. Instead of defining threshold performance values (such 
as “seven attempts to initiate the call for fire is an acceptable value”) the project would 
develop or select a system that would define what should be measured, e.g., “attempts to 
initiate the call for fire.” This would then lead to tests to measure these values, but leave 
it to the operators to decide if it met their needs. 

Modeling and simulation tools were selected to evaluate the performance of the 
alternatives. Vitech’s CORE, Rockwell Automation's Arena and POW-ER (Project, 
Organization, Work for Edge Research) developed through the Virtual Design Team 
research at Stanford University, were used to simulate the operation of the five 
alternatives. CORE modeled both function and data flows; Arena quantified timing and 
assessed the effects of varying inputs; and POW-ER assessed processes by quantifying 
direct work, rework, coordination tasks, decision wait time, and worker backlog. The use 
of these simulation tools to complement and calibrate each other in support of systems 
engineering is a unique approach not found in the existing literature. Performance 
models were created and validated against historical data for one of the alternatives. All 
alternatives were modeled based on the validated model. This provided a high level of 
confidence in their projected performance values. 

A life cycle cost model was generated for each alternative. The historical cost of 
one of the alternatives, based on fully-burdened labor rates and actual labor hours, was 
used as a baseline. A Life Cycle Cost Estimate (LCCE) was created for each alternative, 
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which allowed comparison of overall cost, as well development, operating, and 
retirement cost comparisons. 

An Analysis of Alternatives was conducted to compare the estimated performance 
of each alternative, and its associated cost, to the other alternatives. SME consultation 
was conducted to estimate the performance of the alternatives in some areas. SME 
consultation was also used to validate the functions used to convert raw performance 
evaluation scores into weighted performance scores for the alternatives. 

A comparison of alternatives based on the modeling and simulation results 
indicated the JTEM CTM was projected to have slightly better performance than other 
considered alternatives, and had the median LCCE of the five alternatives. 

The overall recommendation based on this study is to monitor the development of 
the JTEM CTM for further maturation as this project promises significant improvements 
in the overall utility of C4I SoS evaluations. The JTEM CTM was initiated in 2005; the 
cost for final development (2007 through 2009) is estimated at $3.5M, and will be borne 
by the JTEM program. The estimated operating cost, which would be borne by C4I test 
organizations implementing the JTEM CTM, is expected to be significantly lower than 
other alternatives investigated. Changes or uncertainties in performance or costs 
associated with the JTEM CTM will require additional analysis to confirm the expected 
benefits of this approach. Detailed investigation of the JTEM CTM in its entirety, and 
optimization of personnel resources and organizations with a modeling tool such as 
POW-ER, should be pursued. The JTEM CTM should also be used to conduct a C4I SoS 
evaluation as soon as feasible to validate methods and processes in a “real world” event. 
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I. 


INTRODUCTION 


A. BACKGROUND 

Commanders must be able to utilize Command and Control (C2) techniques to 
manage the battlespace under their command. “C2 is essentially about information: 
getting it, judging its value, processing it into useful form, acting on it, and sharing it with 
others” [Joint Pub 6-02, 2006:1-2], The tasks of gathering, judging, and processing 
information have been improved through the creation of Command, Control, 
Communication, Computers and Intelligence (C4I) systems. The intent of C4I systems is 
to provide commanders with the tools necessary to increase their ability to process data 
and increase the tempo of warfighting. 

Over time, warfighting has grown in complexity, tempo, and scope; 
correspondingly, our military must be able to respond with increased agility across 
greater distances. Furthermore, warfighting will continue to change in the future, 
requiring our military to continually adapt and stay ahead in the rapidly shifting global 
environment. Lieutenant Colonel Bemd Horn, Canadian Forces, affirms this position, 

To state that the battlespace of the future - the land, air, sea, space and 
electromagnetic realm where armed conflict will be conducted within its 
cultural, economic, ecological, environmental, political, social and 
technological contexts - will be dramatically different from that of today 
is to repeat the strikingly obvious [Horn, 2003: 8], 

To combat adversaries of today and tomorrow, US forces are joined not only 
across Services but with our international coalition partners to fight enemies as a united 
front. Joint C4I systems, therefore, are essential to the success of our military forces now 
and in the future. The US Joint Chiefs of Staff maintain “Interoperability is key to the 
joint force gaining information superiority in today’s network enabled environment” 
[Joint Pub 6-02, 2006:1-8]. The interoperability of C4I systems is critical to military 
success. 

While the need for interoperable C4I systems has long been recognized, these 

systems have been developed as solutions to separate problems. The services historically 

developed C4I systems in isolation; each system designed to address a need within one 
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warfighting function. As the component C4I systems have developed, the ability of 
multiple systems to collectively achieve capabilities beyond the reach of a single system 
was also recognized. Systems were connected to each other, integrated with each other, 
and assembled in an interdependent System of Systems (SoS). 

The Predator Unmanned Aerial Vehicle (UAV) was originally designed as a non- 
lethal high altitude reconnaissance asset [Airforce Technology, 2007]. The Helicopter 
Launched Fire and Forget Missile (Hellfire) was designed in the 1970s as an air-to- 
ground anti-armor missile to be launched from low-flying Army and Marine Corps 
helicopters [Boeing, 2007]. In 2000, the UAV and the missile, with separate missions, 
requirements, and service owners, went through a “shotgun wedding” and became an 
armed reconnaissance and interdiction SoS, culminating in a successful launch of a 
Hellfire from a Predator in 2001 [Checkpoint, 2007], These two, formerly completely 
independent systems, now are linked; changes to missile weight or range, for example, 
must be considered in terms of effects on the payload and range of the aircraft. 

After-the-fact integration is not unique to weapon systems; the C4I SoS also 
contains systems developed for one purpose, but now used for another. Position Location 
Reporting System (PLRS) was originally designed to provide friendly position location 
information (PLI), identification, and navigation aides. Enhanced PLRS (EPLRS) is now 
used by Army and Marine Corps for digital network communications. The Marines have 
developed C2 On the move Digital Over-the-horizon Relay (CONDOR), which uses 
EPLRS to extend battlefield communications via military satellite communications and 
provide over-the-horizon and on-the-move battlefield connectivity [Armed Forces 
Communications and Electronics Association, 2004], Like the weaponization of the 
Predator, CONDOR integrated separately designed systems into a new SoS, and changes 
to components (EPLRS and military satellite communication systems) must be 
considered in terms of effects on the CONDOR. 

CONDOR is an example of the growth, in an almost biological manner, of C4I 

SoS; new technology continues to be integrated with existing systems without an 

overarching architecture, engineering design, set of performance requirements, or 

managers. Without an overarching view of the SoS, the effects of system-level changes 
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to performance, capabilities, or standards compliance on the SoS are impossible to assess 
in an objective, repeatable, and measurable manner. 


Architecting C4I SoS upfront as a coherent whole, top down design represents a 
shift in this philosophy, as seen in Figure 1. However, it is improbable that the “system¬ 
centric,” bottom up design, approach is likely to cease in the near future. Capabilities- 
based acquisition and integration requires interoperability evaluation and certification 
based on delivering a war-fighter capability as part of a joint SoS. 
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Figure 1. Transitioning Requirements Generation Philosophy 

This figure represents the transitioning philosophy from stovepiped acquisition practices 
to acquisition of systems built from joint capabilities with the joint fighter in mind [After 
Hoivik, 2007: 5], 

The Joint Capabilities Integration and Development System (JCIDS) process 
identifies “.. .the capabilities (and operational performance criteria) required to 
successfully execute missions; the shortfalls in existing weapon systems to deliver those 
capabilities and the associated operational risks; and the possible solution space for the 
capability shortfalls” [CJCSI 3170.OIF, 2007:2], JCIDS is the long term Department of 
Defense (DoD) answer to developing systems that are integrated and interoperable from 
the start. 
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The C4I SoS, however, continues to evolve as urgent unfunded needs statements, 
urgent warfighter needs, and technologies are discovered or developed. These changes 
are implemented as new or extended capabilities to the C4I SoS, which makes the need 
for a system to define performance measures an enduring one. 

Congressman Jim Saxton of New Jersey, member of the Joint Economic and 
House Armed Service Committees, clearly agrees that problems exist with the current 
acquisition practices. 

Presently, the department (of Defense) allows its individual military 
services, agencies and field activities to determine their own IT 
[Information Technology] needs. This stove-piped approach has led to the 
confusing and complex C4ISR (Command, Control, Communication, 
Computers, Intelligence, Surveillance, and Recognizance) landscape that 
exists today. This environment has resulted in waste, redundancy and lack 
of interoperability in IT systems and capabilities for our warfighters 
[Saxton, 2003]. 

Today’s Joint C4I SoS are required to transport and process shared data 
throughout the operating forces. The Joint Communication System doctrine elucidates 
this point. 

A joint force that is linked and synchronized in time and purpose is 
considered networked. ... To do this, the communications system must be 
interoperable, agile, trusted, and shared. Problems are abundant because 
there is no baseline, standard configuration, or overall management of the 
SoS [Joint Pub 6-02, 2006: viii]. 

It is this lack of baselines, standard configurations, and overall management of 
SoS that make defining C4I SoS capability performance measures so difficult. Without 
performance measures for the C4I SoS, however, accurate fielding decisions for SoS 
components cannot be made. Some specific examples of the negative consequences of 
system-focused fielding decisions on SoS capabilities are provided in Chapter I, Section 
C. 


C4I system-level acquisition, testing, and management are well understood, and 
individual systems have documented performance requirements (e.g., Capabilities 
Development Documents or Operational Requirements Documents). Processes and 
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methods for designing and executing C4I system tests are well defined and executed, but 
testing methods at the SoS level are not well defined, nor are consistent standards and 
practices applied. A complicating factor is that real instances of the C4I SoS have a 
practically infinite number of possible configurations. 

C4I SoS have ever-changing configurations that do not have formally established 
performance requirements, nor standard SoS capabilities that can be used to generate 
performance evaluation measures. There is not a clear understanding of how to manage 
or assess C4I SoS performance or C4I SoS capability to support joint or single Service 
missions. 

The Joint Interoperability Test Command (JITC) is the sole DoD certification 
authority for joint and combined interoperability [CJCSI 6212.01D, 2007: 5], JITC tests 
the interoperability of systems, but this only proves that system interfaces function. 

There is no agency that assesses the capability of a SoS to accomplish a task that requires 
the coordinated, successful integration of functions and interfaces across multiple 
systems. Indeed, JCIDS defines interoperability to include “both the technical exchange 
of information and the end-to-end operational effectiveness of that exchanged 
information as required for mission accomplishment” [CJCSI 3170.OIF, 2007]. 

The goal of this project was to design a Joint C4I Capability Certification 
Measures (JC3M) system to fill the current capability gap that exists in the determination 
of a system’s effect on SoS performance. JC3M is important because it is intended to 
provide SoS users, as well as the acquisition community, with much-needed performance 
requirements for the design of new systems, integration of legacy systems, and SoS 
testing. JC3M is also important because it will provide system integrators a tool to assess 
integration formally, it will document system capabilities and construction, and it will 
provide confidence to the warfighter that the C4I SoS works. C4I SoS have been custom 
built to date, with all the Configuration Management (CM), troubleshooting, training, and 
support challenges this “one-off’ approach implies. With a consistent assessment 
methodology and a documented baseline configuration, C4I SoS evaluation becomes 
repeatable. This repeatability allows CM, training, troubleshooting, and knowledge 
management costs to shrink. 
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The JC3M team defined a system for use by C4I test organizations that will 
identify the desired war fighting capabilities of the C4I SoS and ensure that the system 
under test (SUT) meets these requirements. The JC3M system included Doctrine, 
Organization, Training, Materiel, Leadership, Personnel, and Facilities (DOTMLPF) 
considerations in designing the system to be used by an organization, following 
repeatable processes, and using consistent resources. 

Parallel efforts are underway to address SoS evaluation, and these approaches 
were considered along with other candidate solutions for JC3M. Existing efforts include 
the JTEM CTM, which is addressing Joint SoS performance evaluation from the Office 
of Secretary of Defense (OSD) level. Marine Corps Systems Command 
(MARCORSYSCOM), the acquisition organization for the Marine Corps, is approaching 
the issue from a Service perspective. MARCORSYSCOM has tasked Marine Corps 
Tactical Systems Support Activity (MCTSSA) to develop MC3T, a methodology for 
managing the MAGTF C4I SoS as a single system, in accord with modern systems 
engineering practices. MC3T will manage the MAGTF C4I SoS as a set of SoS-level 
capabilities, rather than as a fixed hardware or software baseline. 

JC3M leveraged MC3T and JTEM efforts, cooperating and consulting with both 
organizations as stakeholders. Individual JC3M team members will work with both 
JTEM and MC3T as their processes mature and are implemented. JTEM is expected to 
complete development in 2009; MC3T is conducting a proof of concept event from 
October through December 2007. 

B. PROBLEM STATEMENT 

Information Age C4I systems have grown biologically into an interdependent 
entity, a System of Systems. There is no clear consensus on how to evaluate a C4I SoS 
capability, that is, a task only achievable by interdependent multiple components. Current 
inter-operability certification processes only partially address SoS end-to-end evaluation; 
systems are certified and delivered with interfaces functioning from a technical 
perspective but with less than optimum results or end-user dissatisfaction from an 
operational effectiveness standpoint. Capabilities-based acquisition requires 
interoperability evaluation and certification based on delivering a war-fighter capability 
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in an operational environment as part of the SoS. The primitive need, as identified by the 
MC3T team lead, was for testable threshold values to use in the design and conduct of a 
C4I SoS test. This statement is based on experience at MCTSSA in conducting SoS tests 
for acquisition agencies. This led the team to the problem statement below: 

There is no system that defines and compares System of Systems 

performance measures to warfighter needs in an objective and measurable 

way [Finn, 2007], 

C. OPERATING FORCES NEED FOR JC3M 

The lack of SoS-level performance measures is not an issue that only affects the 
test community; there are consequences to the operating forces as well. Two examples of 
system-level changes which were tested and fielded without SoS-level performance 
measurements are provided. The test cases are summarized below. 

1. Example 1 - Voice Switch Fielding 

The first example, testing and fielding of voice switches, affected operating forces 
communication networks. This problem involves three different voice switches fielded 
over eighteen years for the Marine Corps. A voice switch is a system of electronic 
components that connects telephone calls, providing voice communications between 
calling and called subscribers. When placing a telephone call, the person initiating the 
communications is a calling subscriber and person receiving the communications is a 
called subscriber. When a calling subscriber makes a phone call to a called subscriber, the 
calling subscriber hears a ring back tone (ring ... ring, in the ear piece) and the called 
subscriber’s phone physically rings. The called subscriber picks up the phone and the 
voice communication path is established. This communication capability is made 
available through the use of voice switches. 

The Unit Level Circuit Switch (ULCS), Integrated Serviced Digital Network 
(ISDN) Gateway Exchange (IGX), and Compact Digital Switch (CDS) are switches 
interoperated through trunks which provide voice paths between two switches. Three 
different types of telephones were involved: Secure Terminal Equipment (STE) in Plain 
Old Telephone System (POTS) mode for non secure calls, Digital Secure Voice Terminal 
(DSVT) for secure calls and Digital Non-secure Voice Terminal (DNVT) for non secure 
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calls. The STE was connected to IGX; the DSVT and DNVT phones were connected to 
the CDS and ULCS 

Each voice switch is similar, and provides secure and non secure voice 
communication capabilities. CDS and ULCS switches are designed and built to the tri¬ 
service tactical (TRITAC) communication specifications. The IGX was required to 
interface with both the CDS and ULCS by implementing a specification the same as the 
TRITAC specification. The architecture can be configured in many ways forming the 
SoS that provides a voice communication capability 

Testing was conducted and a voice call was successfully completed between IGX 
and CDS using a non secure telephone. The conclusion was that the IGX and CDS 
worked according to the system-level specifications. Voice calls were successfully 
completed between CDS and ULCS using all three different telephones. The conclusion 
was that CDS and ULCS worked according to the system-level specifications. After 
testing, the systems were fielded. 

After fielding, the operating forces implemented an architecture using all three 
switches connected in series; the IGX was connected to the CDS, which was in turn 
connected to the ULCS. Users found a call could not be successfully completed from a 
non secure phone from the IGX through the CDS to a DNVT called subscriber at ULCS 
switch. The called subscriber DNVT rang but the calling subscriber’s non secure phone 
did not get a ring back tone. The interoperability and specifications had been tested, and 
all three systems passed. The problem was that the architecture that supported the 
capability, as implemented by the user, was not tested. As a result, these switches were 
fielded, and it was only after delivery to the operating forces that this failure was 
observed. This failure continues to this day, forcing workarounds for communicators, 
and limiting their use of voice switches in all configurations. 

The JC3M system would have identified the SoS architecture(s) that provide the 
capability, including viable configurations, tested those configurations, and ensured the 
configuration was documented for use by the operating forces. Without JC3M, however, 
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these switches were fielded and the operating forces discovered after fielding specific 
configurations did not work. 

2. Example 2 - Gateway Exchange Interoperability 

The second example, gateway exchange interoperability, also affected operating 
forces communication networks. The ISDN IGX and High Density Exchange (HDX) are 
voice switches with a Radio Wireline Interface (RWI) Circuit Card Assembly (CCA) that 
provides interface capability with radios. Single Channel Ground to Air Radio System 
(SINCGARS) radios provide wireless voice communications. IGX , HDX and 
SINCGARS are assembled as a SoS to provide voice communications capability to 
subscribers through voice switches and radio operators. 

There were three different versions of software used in the IGX and HDX, and 
testing was performed on all three versions. The configuration of the SINCGARS and 
RWI CCA did not change. 

The version interoperability between IGX version 6.0a and SINCGARS radio was 
successful, but IGX version 6.1a and HDX version 1.0a were not successful. When 
versions 6.1a and HDX 1.0a were developed, they were not tested with the RWI CCA. 

The JC3M system would have identified the SoS architecture(s) that provides the 
capability, to include the legacy system hardware, the RWI CCA, with HDX. During 
SoS testing, the interoperability problem between voice switches and SINCGARS radios 
would have been identified and fixed, before it was fielded. Although the RWI CCA is at 
the end of its life cycle, it is part of HDX, the latest voice switch. Had there been a JC3M 
system, there would have been a documented requirement for RWI interface with IGX 
and HDX prior to testing 

D. THE SYSTEMS ENGINEERING DESIGN PROCESS 

1. Overview 

The team implemented a Systems Engineering (SE) approach that started with the 
identification of customers’ needs and proceeds through the phases illustrated in Figure 2 
until a recommended solution was generated. Each phase was broken down into one or 
more subtasks that break the larger phases into elements. At the end of each phase the 
opportunity to re-evaluate progress and return to a prior stage for refinements was 
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available if necessary. This refinement process was critical to adapting the project to 
ensure the customers’ and stakeholders’ needs are achieved. The SE process model 
below was based on a combination of System Engineering Design Process (SEDP) [Paulo 
(a), 2005] and State the problem, Investigate alternatives, Model the system, Integrate, 
Launch the system, Assess performance, and Re-evaluate (SIMILAR), modified and 
combined to fit this project. 

The SE process presented in Figure 2 is slightly different than the original SE 
process provided within the Project Management Plan (PMP) (Appendix A). In the 
original SE process, the Modeling phase composed a single phase and the Simulation and 
Analysis of Alternatives (AoA) elements were joined in the next phase. Through the 
implementation of the SE process, the team realized that this did not truly represent the 
phases as they were logically and physically executed. Therefore, the simulation element 
was moved to the modeling phase. The result is a phase boundary change, and not a 
reorganization of the SE process. 



Figure 2. JC3M Systems Engineering Process. 

This SE process model is based on combination of SEDP by Professor Paulo and 
SIMILAR System Engineering Process Model from INCOSE. Central to the JC3M 
Systems Engineering Process is the ability to re-evaluate after each element to update or 
rework an element as required prior to moving on. 
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2. JC3M Systems Engineering Process Phases 

Each of the JC3M Systems Engineering process phases is briefly described to 
provide a summary of the SE process the team utilized throughout this project. Each 
phase and the associated elements are described in greater detail within Appendix A. 
a. Problem Refinement Phase 

The team utilized the Problem Refinement phase to clarify stakeholders’ 
needs and begin managing the JC3M project risks. The outputs of this phase allowed the 
team to design multiple solutions for the JC3M system problem. The Problem 


Refinement Phase is composed of three key elements: Needs Analysis, Requirements 
Generation and Analysis, and Values System Design, which includes definition of 
Evaluation Measures (EM). Each element and the associated inputs and outputs are 
provided in Figure 3. It was also during the Problem Refinement phase that the team 


initialized risk management techniques. 
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Figure 3. JC3M Problem Refinement Phase. 

The Problem Refinement Phase consists of three key elements, each providing outputs 
for later phases and elements. Central to the JC3M Systems Engineering Process is the 
ability to re-evaluate after each element to update or rework an element as required prior 
to moving on. 
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b. Design Alternatives Phase 

The team utilized the Design Alternatives phase to generate multiple 
candidate solutions to the problem, establish feasibility criteria and apply those criteria to 
eliminate those alternatives that were clearly infeasible. The creation of these solutions 
required the outputs of the Problem Refinement phase. The outputs of this phase were 


used for the creation of models in the Modeling and Simulation phase. The Design 
Alternatives Phase is composed of the three key elements: Alternatives Generation, 
Feasibility Criteria, and Alternative Scoring. Each element and the associated inputs and 


outputs are provided in Figure 4. 


Design Alternatives 



Figure 4. JC3M Design Alternatives Phase. 

The Design Alternatives phase consists of three key elements, each providing outputs for 
later phases and elements. 
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c. Modeling and Simulation Phase 

The team utilized the Modeling and Simulation phase to generate models 
based on the alternatives selected in the Design Alternatives phase. Once the models 
were built, the alternatives were simulated to provide results for comparison in the AoA 
phase. The Modeling and Simulation phase is composed of the two key elements: Model 
Development and Simulation. Each element and the associated inputs and outputs are 


provided in Figure 5. 


Modeling and Simulation 
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Output: 
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Outputs: 

* Simulation Results 

* Cost Data 


Figure 5. JC3M Modeling and Simulation Phase. 

The Modeling and Simulation phase consists of two key elements, each providing outputs 
for later phases and elements. 
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d. Analysis of Alternatives Phase 

The team utilized the Analysis of Alternative phase to examine each 


alternative’s cost in light of its performance that resulted from the Modeling and 
Simulation phase. Multi-attribute value modeling techniques were employed to evaluate 
and rank each of the alternatives. The output of this phase provided utility scores and life 
cycle cost estimates for each alternative to the Final Recommendation phase for decision 
making. The Analysis of Alternatives Phase is composed of the two key elements: Cost 
Analysis and Multi-attribute Value Modeling. Each element and the associated inputs 
and outputs are provided in Figure 6. 
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Figure 6. JC3M Analysis of Alternatives Phase. 

The Analysis of Alternatives phase consists of two key elements, each providing outputs 
for later phases and elements. 
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e. Final Recommendation Phase 

The team utilized the Final Recommendation phase to assemble all earlier 
inputs, leading to a preferred alternative and a cohesive recommendation for 
implementation, which is published in this final report. The output of the Final 
Recommendation Phase will be provided to both the JTEM and MC3T teams for their 
use. The MC3T implementation team is a critical stakeholder, but their schedule includes 
a proof of concept event in October 2007. Based on this schedule, the MC3T team 
concluded they could not wait for JC3M project outputs. MC3T implemented an interim 
process, and will review the JC3M project report for inclusion of applicable 
recommendations as MC3T refines their processes. The Final Recommendation Phase is 
composed of one key element: Recommendation. The element and the associated inputs 
and outputs are provided in Figure 7. 

Final Recommendation 


Recommendation 

Inputs: 

* Utility Vs. Life Cycle Cost 
Estimate 

Output: 

• Stakeholder 
Recommendation 


Figure 7. JC3M Final Recommendation Phase. 

The Final Recommendation phase consists of one key element. This phase is the 
culmination of all the prior phases and provides the recommendation to the stakeholders 
for concurrence. 
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f. Assess Performance Phase 

This project does not completely conclude after delivery of the final 
report. Some members of the team will continue to work with MC3T, ITEM, and other 
stakeholders after this project concludes. Some members of the team already participate 
in the JTEM Capability Testing Community of Interest, where they will share their 
experiences with other members and continue to advance the art of capability testing. 
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II. PROBLEM REFINEMENT 


Problem Refinement was the stage of the JC3M process that refined the initial 
understanding of the problem into a workable system design. The JC3M SE process, 
discussed in Chapter I, was used in an iterative manner both to direct the process of 
developing a solution, as well as to guide the team away from immediately designing the 
JC3M solution before all the requirements were understood. Each stage of the SE 
process was iterative; each phase and element presented the opportunity to re-evaluate 
progress and return to a prior stage for refinement if necessary. While this presented a 
challenge in managing the growth of the project (“scope creep”), it was also critical to 
adapting the project to ensure the needs of the stakeholders as identified in this Chapter 
were met. 

To accomplish the transformation from perceived need to documented 
requirements, the SEDP [Paulo (a), 2005] calls for Stakeholder Analysis, Input-Output 
Modeling, and Functional Decomposition. The JC3M team blended the SEDP 
methodology with a modified customer requirements capture process [Stevens et al., 
1998: 28] and separately collected user and system requirements, organized the 
requirements, and created requirements documentation. 

Problem Refinement consisted of three tasks: Needs Analysis, Requirement 
Generation and Analysis, and Value System Design. Needs Analysis task identified 
stakeholders relevant to the problem. While some [Whitten and Bentley, 2007:7; Stevens 
et al, 1998:21] define stakeholders as only persons, the JC3M team chose to include 
selected organizations as stakeholders because the duration of organizations and the 
missions those organizations perform outlast individuals. MCTSSA has been testing C2 
software since 1972, yet none of the original MCTSSA personnel remain at the 
organization. 

The Needs Analysis portion of the SEDP conducted an investigation of the 
problem, in order to refine the problem from a primitive need to an effective need, and 
deliver a refined problem statement. Needs Analysis also served as a control gate, which 
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rendered a go or no-go decision for continuation of the design process. If the Revised 
Problem Statement, the primary output of Needs Analysis, had revealed the effective 
need was already met, thus stopping the design process, the JC3M team would have 
undertaken a different project. Because the Refined Problem Statement showed an 
effective need existed, it provided justification to continue further with the design 
process. 

Requirements Generation and Analysis took the Refined Problem Statement and 
performed system decomposition to clarify requirements. Those requirements were 
decomposed into critical functions and functional flows. The outputs of Requirements 
Generation and Analysis were the Problem Refinement Report and the JC3M Functional 
and Value Hierarchies. 

Value System Design took the Problem Refinement Report and Functional 
Hierarchy and identified characteristics and measurable parameters of the JC3M system, 
which were used later to evaluate alternative solutions. Outputs of Value System Design 
were Evaluation Measures (EM) as well as the JC3M values and functions, documented 
in the JC3M Value Hierarchy 
A. NEEDS ANALYSIS 

Needs Analysis consists of identifying stakeholders and their requirements 
[Verma, 2006], The functional need identified by stakeholders was translated into 
qualitative and quantitative requirements later in the JC3M design process. 

1. Stakeholder Identification 

Key stakeholders must be identified in order to determine their requirements. 
Whitten and Bentley [2007] and Stevens [1998] described the need for stakeholder input, 
but assumed stakeholders were identified. Verma [2006: 8] stipulates that stakeholders 
can be active or passive, yet does not explain how internal or external stakeholders can be 
identified. Identification of the people and organizations that were to become JC3M 
stakeholders was critical, so the team used several methods in this stage. 

First, the team created a list of potential stakeholders. An internal review of the 
team confirmed a substantial experience base in C4I SoS testing in general, current test 

processes, and the development of new test processes. Team members were experienced 

18 



in hardware and systems testing; systems development and testing; C4I SoS testing; and 
the development of new test methodologies. This experience base was polled, and a list 
of potential stakeholder organizations was created: 

■ Joint Interoperability Test Command, Ft. Huachuca, AZ 

■ Marine Corps Operational Test and Evaluation Activity (MCOTEA), 
Quantico, VA 

■ U.S. Army Test and Evaluation Command, Alexandria, VA 

■ U.S. Navy Operational Test and Evaluation Force, Norfolk, VA 

■ U.S. Air Force Operational Test and Evaluation Center, Kirtland AFB, 

NM 

■ MCTSSA, Camp Pendleton, CA 

■ U.S. Army Program Executive Office for Command, Control, and 
Communications Tactical (PEO C3T) 

As the project continued, some members were assigned, as part of their normal, 
non-academic duties, to investigate and develop a new test methodology (see Chapter III, 
Section A.l. Known Alternatives - MC3T). As part of this effort, external to JC3M, the 
team was introduced to the JTEM Capability Test Methodology (CTM) process. The 
JTEM CTM team proved to be a very valuable participant in defining requirements, 
validating key JC3M concepts and service, and an alternative JC3M solution (see Chapter 
III section A.l Known Alternatives - JTEM CTM for details). 

The team refined the list of potential stakeholders, targeting organizations that 
were interested, willing to participate, and representative of a portion of the C4I test 
community. The team identified organizations that provide service-specific test 
perspective on current test challenges (MCTSSA, PEO C3T); DoD-level perspective on 
current and future test developments (JITC, JTEM CTM); and academic and theoretical 
perspective (NPS). The key stakeholders that were consulted throughout the duration of 
the project were: JITC, MCTSSA, JTEM CTM, PEO C3T, and NPS. 

2. Stakeholder Initial Requirements 

As stakeholders were identified, perceived needs were reviewed and documented 
in an initial problem statement. The primitive need was for testable threshold values to 
be used in the design and conduct of a C4I SoS test. While Finn [2007] defined what he 
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perceived as the effective need, a system to evaluate SoS performance, Bjorkman 
[(a),2007] agreed with the team that the definition of test criteria is a central challenge in 
SoS testing. SoS complexity and test challenges are not limited to DoD, but in the 
interests of managing the scope of the project, the JC3M team will focus on DoD C4I 
SoS. 

B. REQUIREMENTS GENERATION AND ANALYSIS 

Requirement Generation and Analysis was another critical stage in the design 
process, because it further clarified requirements, identified critical functions of the 
system, and documented those requirements for use in all later stages. 

1. Requirements Identification 

The goal of requirements identification was to move from the primitive need 
expressed above to a more refined, more accurate effective need. The primitive need is 
reflective of the stakeholder’s bias, experience, and perceptions [Paulo (c), 2005:1] so the 
JC3M team explored this perceived need, and looked at the larger context of the problem, 
to find the true, underlying (effective) need. 

The transition from primitive need to effective need was based on stakeholder 
inputs that were captured via stakeholder focused interviews and were aided by 
questionnaires. After reviewing the primitive need, the JC3M team generated a 
questionnaire to solicit comments, perspective, and specific guidance from potential 
stakeholders. The questionnaire was reviewed verbally or provided for review 
electronically. The questionnaire and responses are provided as Appendix B. 
Respondents include: 

■ JTEM: verbally reviewed with Dr. Dave Dryer and Mr. Tim Beach on 14 
February 2007 

■ MC3T: reviewed in person with Mr. Ian Finn on 23 February 2007. 

■ Central Technical Support Facility (CTSF): provided in February 2007. 
Comments not released. 

■ JITC: questionnaire provided in February 2007. Comments received 28 
Feb 2007 
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The initial version was formulated to capture SoS information, but did not capture 
information necessary to refine JC3M requirements. The revised questionnaire was 
submitted to JITC, and their response is also included in Appendix B. In their response, 
JITC Subject Matter Experts (SMEs) provided detailed guidance on overarching 
interoperability and integration requirements, DoD mandated events and processes, and 
their role in standards conformance and interoperability testing. This information was 
central to the development of the two new alternative solutions for JC3M. The JITC 
responses also cemented the developing relationship between JITC and the team. JITC 
SMEs were consulted throughout the project, participated in both of the project formal 
reviews, and assisted with the validation of requirements and approaches. 

2. Requirements Analysis 

Neither interviews with stakeholders nor internal team reviews could identify a 
system to satisfy the primitive need as articulated by Finn [2007]. The team determined 
the effective need was for a system that could define threshold performance values for the 
C4I SoS. These threshold performance values could be used, in turn, by existing SoS test 
systems to evaluate SoS performance. Internal and external reviews, as well as a search 
of the existing literature, showed there was not a system in place to meet the effective 
need. 

As a solution to this problem, the JC3M system was to be designed to meet the 
effective need in the planning, conduct, and reporting of C4I SoS performance. Note that 
existing Joint or Service valid processes were used where applicable; the JC3M system 
does not, for example, define how to conduct a stakeholder meeting, define the creation 
of test scripts, or define how to communicate test results to stakeholders. Because the 
conduct of assessments is well-understood and executed by C4I test organizations, as is 
reporting the results of those assessments, the JC3M system is focused on the steps 
leading up to and including planning of an assessment. 

The functional analysis of JC3M requirements was conducted by the JC3M team, 
and the model was validated in discussions with Mr. Ian Finn, MC3T test lead at 
MCTSSA, as well as at JC3M Program Reviews (16 March 2007, 08 June 2007). 
Attendees included representatives from JITC, MCTSSA, PEO C3T, and JTEM. This 
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model was subsequently expanded by the team to develop the value systems hierarchy 
(see Chapter II section C, Value Systems Design). The hierarchy of JC3M was 
constructed based on three major functions in the evaluation of a C4I SoS. The first 
function, Planning the C4I SoS Evaluation, was the focus of JC3M. The second function, 
Conducting the C4I SoS Evaluation, as well as the third function, Reporting the Results 
of the C4I SoS Evaluation, are not modeled, for reasons stated above. 

From the Plan function, the JC3M team decomposed sub-functions (Define 
Problem, Identify Systems, Define Criteria and Plan Review). This analysis identified 
planned inputs and outputs from each function and sub-function. Figure 8 illustrates the 
major functions of JC3M. 



Figure 8. JC3M Functions. 

Development of JC3M focused on Planning a C4I SoS Evaluation. This figure illustrates 
the decomposition of subfunctions within the JC3M Planning Process. Conducting a C4I 
SoS evaluation (2.0) and Reporting the Results of the Evaluation (3.0) were recognized 
as well-functioning processes at C4I test organizations; JC3M proposed utilization of 
these existing processes, rather than developing new processes. 

3. JC3M Inputs and Outputs 

Inputs to the JC3M system include stakeholder requirements, C4I SoS 
components, and labor. Outputs of the JC3M system include the assessment plan for the 
C4I SoS and capabilities under review; EM; and a SE Artifact report, which describes the 
current state of requirements documentation for the SoS and components under review. 
One of the secondary outputs includes a documented C4I SoS configuration with 
documented capabilities, which in turn can provide war-fighters with a known baseline 
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configuration for implementation. If utilized, this baseline increases war-fighter 
confidence in the effectiveness of the C4I SoS, and reduces the resources required for 
C4I SoS implementation, use of the SoS, training, troubleshooting, and life cycle support. 
While the assessment plan is a by-product, the directive portion of the assessment plan 
(“in order to assess the capability to perform Task X... the System Y operator will 
initiate a request for support by sending message 1 via function 2 from screen 3... ”) can 
be used as a “how-to” list for SoS operators. 

C. VALUE SYSTEM DESIGN 

Value System Design was the process used to create a qualitative and quantitative 
model [Paulo (c), 2005] which portrayed the system functions, objectives, and evaluation 
measures. The quantitative model was used to analyze and evaluate the performance of 
the alternative solutions. Value System Design was conducted by Value System 
Modeling and creating a Value Hierarchy. 

The project team found the description “... developing a good set of MOEs 
[Measures of Effectiveness] is usually a harrowing business” [Feuchter, 2000: 40] an 
accurate account of the process. 

1. Value System Modeling 

The value system of JC3M system was constructed based on a three-phase model 
of C4I SoS performance evaluation: Plan the execution of the evaluation, Conduct the 
evaluation, and Report on the evaluation. This model was constructed based on JC3M 
team experience with C4I SoS evaluation. 

From the top-level functions (Plan, Conduct, Report) the JC3M team identified 
sub-functions, and an objective was identified for each of the sub-functions. After 
objectives were established, each was reviewed to determine an appropriate level of 
performance which was used in comparing the alternative solutions [Feuchter, 2000: 39- 
40], The overall goal was to be able to drill down from effective need for JC3M, to get to 
the “what” that JC3M was designed to do, and eventually to the “how well” standards of 
performance. As seen in Figure 9, Feuchter calls the “how well” factors MOE; the JC3M 
team refers to these as EM. 
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Goodness 

Methodology 

Development 



Identify Goals 


Identify Tasks to be Performed 
to Meet Goals 


Identify MOEs that Measure 
How Well Each Alternative 
Performs Each Task 


Figure 9. Developing Evaluation Measures 

Evaluation Measures Trace Back to System Goals. The complete Value System Model 
process and results are included in Appendix E. Most EM have a single quantifiable 
factor, but some have multiple factors. A graphical layout of the Value System Hierarchy 
is also included as Appendix E [From Feuchter, 2000:40]. 

2. Value Hierarchy 

The JC3M team determined functional analysis [Blanchard and Fabrycky, 1998: 
26] was required to ensure stakeholder requirements were incorporated in the final JC3M 
system. Stakeholder requirements were collected (Appendix C - Questionnaires), and a 
value hierarchy [Paulo (b), 2005] was created to translate stakeholder requirements into 
design criteria [Blanchard and Fabrycky, 1998: 29] for the JC3M system. The initial 
decomposition of system requirements is displayed in Figure 10. 



Figure 10. JC3M Top Level Value Hierarchy. 

Top level value hierarchy for the JC3M system after stakeholder input. As described, 
JC3M will focus on the functional requirements in section 1.0, and utilize existing 
processes and methodologies for 2.0 and 3.0. 
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The team developed the value hierarchy with both functional (1.0, 2.0, 3.0) 
requirements and non-functional (4.0, 5.0, and 6.0) requirements [Buede, 2000: 39]. 
From this initial level of requirements, more discrete sub-functions were identified by 
reviewing stakeholder expressed and implied requirements. The complete JC3M value 
hierarchy and all measures are provided at Appendix E. Figure 11 shows the critical 
measures in white boxes. 



Figure 11. JC3M Functional Hierarchy. 

This figure illustrates selected JC3M subfunctions based on the JC3M Value Hierarchy 
(illustrated Figure 10). Figure 11 also illustrates the traceability of EM to JC3M 
functional and non-functional requirements. 

3. Critical Evaluation Measures 

EM were developed for each function of the JC3M system, to ensure each 
function was not only measurable, but potentially capable of serving as a discriminator in 
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considering alternative JC3M solutions. This section provides a summary of the critical 
EM to be used in analyzing JC3M alternative solutions. A detailed description of all 
JC3M EM is provided in Appendix E 

Table 1 identifies the critical EM used to compare the performance of each 
alternative; cost was calculated separately, and later compared to the overall performance 
of each alternative. The body of the table contains the definition of each evaluation 
measure, which describes how the EM was calculated. The range and format of expected 
values of the EM are defined, and the rationale and relevance describes what the EM 


measured, and why the EM was used to compare alternatives. 



Percentage of 
Traceable 
Measures 

Days to 

Plan 

Evaluation 

Quality of 

Planning 

Products 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

JC3M 

Function 

Define Measures 
1.3.2 

Planning 

Results 

1.4.3 

Planning 

Results 

1.4.3 

Input System 
Flexibility 

4.1 

Input System 
Flexibility 

4.1 

Definition 

Alternative 

generated 

measures, traceable 
to stakeholder 
requirements, 
divided by the 
number of 
measures generated 
by the alternative. 

Elapsed time 
(in days) of 
planning for 

C4I SoS 
evaluation 

Assign an 
overall quality 
level to the 
planning 
documents 
produced. 

Divide percent 
change in labor 
hours to conduct 
planning phase 
of JC3M by the 
percent change 
in systems under 
test. 

Divide percent 
change in duration 
to conduct 
planning phase of 
JC3M by the 
percent change in 
systems under 
test. 


Ratio level data, 
from 0 - 100% 

Ratio level data 
> 0 days 

Ratio level data 
from 1-4 

Ratio level data 
from 0 - oo 

Ratio level data 
from 0 - oo 

Rationale 

and 

Relevance 

Identifies 
objectivity of 
performance 
measures. 

Predicts SoS 
evaluations that 
can be 

conducted in a 

Identifies 
predicted 
utility of 
alternative. 

Predicts changes 
in cost of SoS 
evaluation based 
on size. 

Predicts changes 
in duration of SoS 
evaluation based 
on size. 


Performance 
measures traceable 
to doctrine will be 
perceived as 
objective, 
increasing the 
value of the 
evaluation. 

year. 

Alternatives 
that permit 
multiple SoS 
evaluations 
generate data to 
support fielding 
decisions 

sooner. 

Quality of the 
planning 
products drives 
the overall 
value of the 
alternative. 

Can be used to 
determine most 
effective 
alternative based 
on SoS size. 

Can be used to 
determine most 
effective 
alternative based 
on SoS size. 


Table 1. JC3M Critical Evaluation Measures. 

This table illustrates Critical EM used to compare the performance of alternative 
solutions. Dimension of the measure (Nominal, Ordinal, Interval, or Ratio-level data) is 
provided. Rationale for, and relevance of, the EM is also provided. Note “Quality of 
Planning Products” is expressed on a Likert Scale as described in Appendix C. 
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III. DESIGN ALTERNATIVES 


The goal of the Design Alternatives phase was to provide possible JC3M 
solutions that were broad enough in approach to present stakeholders with a viable range 
of approaches to the problem, yet few enough in number so as to be capable of analysis 
within project constraints. The Design Alternatives phase consisted of three tasks: 
Alternative Generation, Feasibility Screening, and establishing Criteria for Scoring. 

Alternative Generation took the outputs from the Problem Refinement phase, 
including the problem refinement report and JC3M value hierarchy with evaluation 
measures, and created a range of alternative solutions to focus on solving the JC3M 
problem statement - identifying objective threshold values for measuring SoS 
performance. Alternative Generation also included the consideration of known 
alternatives: the status quo, a near-term future solution, and a long-term future solution. 
With three known alternatives, the team generated two new alternatives to present a 
sufficiently broad range, yet still maintain the ability to conduct a thorough analysis. 

Feasibility Screening was used to evaluate the variety of alternatives, and identify 
those which appeared to be feasible, different from the baseline and other alternatives, 
and few enough in number to be effectively analyzed within the scope of the project. The 
output of this process was a smaller set of feasible alternatives. 

Criteria for Scoring were established based on EM, problem refinement report, 
and the JC3M value hierarchy tree in order to identify those criteria which would be later 
used for comparing the alternatives. This process also identified those EM which would 
be drawn from the simulations of the alternatives. Outputs of this process included the 
scoring criteria identified, as well as a blank quantitative modeling decision matrix. 

A. ALTERNATIVE GENERATION 

The team identified known alternatives for inclusion in the set of possible JC3M 
solutions by reviewing historical processes, interviewing current stakeholders, and 
participating in new system development. 
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1. Known Alternatives 

a. Federation of Systems 

The first known alternative was the FEDeration Of Systems (FEDOS) 
process used at MCTSSA. FEDOS was a user-driven process, with C4I program 
managers collectively defining the configuration of the C4I SoS [Manning (c), 2007], 

FEDOS was designed to assess the performance of C4I systems when 
assembled into the Marine Air Ground Task Force (MAGTF) C4I SoS. FEDOS began at 
the order of the Deputy Commander for C4I Integration and Interoperability (C4II) at 
MARCORSYSCOM, who tasked MCTSSA to assess MAGTF C4I SoS and systems 
interoperability. FEDOS was created because there were processes to assess the 
interoperability and performance of a C4I system, but none to assess the performance or 
interoperability of the C4I SoS. The Deputy Commander for C4II assembled a working 
group of stakeholders from the C4I system community at MARCORSYSCOM, and 
tasked MCTSSA to assign a test director. The working group decided what systems 
would participate, what requirements were to be tested, and the schedule of events to 
include test planning, test conduct, and results reporting. 

Because FEDOS provided an opportunity to evaluate the performance of 
C4I systems in the SoS, it was also used to evaluate performance of developmental 
systems. CONDOR, as an example, was developed to provide communications 
connectivity to units physically distant from stationary long haul communications paths. 
CONDOR was included in a June 2005 FEDOS event, prior to deployment to Iraq in 
September 2005. 

The MARCORSYSCOM Product Groups, responsible for developing, 
fielding, and supporting C4I systems, were not required by order or doctrine to 
participate in FEDOS, and a passing grade from FEDOS was not a requirement for a 
milestone decision. This led to the perception of FEDOS as a cost, schedule, and 
performance risk, which in turn led to ineffective participation from acquisition 
community representatives. 
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Because the MAGTF C4I SoS was not designed in compliance with 
architecture, there are no overarching C4I SoS performance measures or threshold 
criteria. This lack of doctrinal C4I SoS performance criteria meant that MCTSSA test 
personnel had to engage in long and at times inconclusive negotiations with stakeholders 
in order to define the threshold values that were used to measure SoS performance, and 
determine if components passed or failed the test. The Product Groups viewed the 
results of FEDOS askance, as the event results were not tied to objective SoS-level 
performance requirements. 

Further, FEDOS was perceived as a no-win situation for Product Groups. 
After a system had successfully passed Development and Operational Tests, and 
demonstrated compliance with system-level performance requirements as described in 
the Capability Development Document (CDD), Operational Requirements Document 
(ORD), or equivalent, FEDOS tested component systems in ways they had not been 
designed to be used. 

The conduct of FEDOS, outlined in Figure 12, was organized in a series of 
events, each of which culminated in a decision point known as a “Control Gate.” Each 
gate represented a milestone that had to be reached before the planning effort could 
continue. The gates included agreements on what was under test, how the network would 
be configured, how the systems would be evaluated, what specific test procedures would 
be used, interpretation of results, and the contents of the test plan. 

Both execution and reporting of SoS evaluation results have effective 
processes in place; JC3M focuses on the planning of SoS evaluations. For the purposes 
of this project, only the initial, Planning phase of FEDOS was considered as a candidate 
alternative JC3M solution. This phase consisted of three tasks as discussed below. 

In Task 1, the team elicited test requirements. This was done by analysis 
of stakeholder inputs and identified hardware and software versions, test schedule 
requirements, and functional test requirements. The functional requirements were 
determined based on the SoS capabilities that were to be exercised in order to evaluate 
the SoS as requested. Additionally, documentation and data collection requirements were 
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identified, as well as customer communications requirements. After stakeholder review, 
changes were incorporated into a final list of test requirements to be presented at the 
control gate exit from Task 1. This requirements list included a draft SoS components 
list, draft project schedule, draft functional requirements and architecture, draft 
documentation and data collection plan, and draft customer communications plan. Figure 
12 provides an illustration of the serial processes in FEDOS. Figure 13 provides a 
detailed view of FEDOS Task 1. 

Task 2 generated an evaluation plan. The FEDOS team reviewed Task 1 
outputs and generated draft Test Thread Architecture diagrams, and reviewed with 
stakeholders. As stakeholder approval was received for the test thread architecture, draft 
test thread descriptions were generated. Draft metrics and a grading process were 
generated and reviewed with stakeholders. Finally, a draft Evaluation Plan, containing 
the draft architecture diagrams, test thread descriptions, and grading process was 
assembled and reviewed with stakeholders. After review, incorporation of requested 
changes, and assembly of the final evaluation plan, the stakeholders had to agree to the 
evaluation plan as another control gate. 

Task 3 generated a test plan and test procedures. The team created a draft 
test plan, incorporating outputs from Task 1 (requirements list) and Task 2 (evaluation 
plan). A test procedure format was chosen, reviewed with stakeholders, and used to 
document the test procedures proposed for the event. Both the test plan and procedures 
were reviewed in detail and presented to stakeholders for approval. Once the test plan 
and the test procedures were approved, the planning phase of FEDOS was concluded. 

Because FEDOS was the only alternative solution that had been used by a 
C4I test organization, it was considered the “status quo” or baseline JC3M alternative 
solution. 
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Output 


7 High Level Tasks to 
complete a test event 


Control Gate 

# 


Elicit Test 
Requirements 
TASK #1 


Hardware/Software Version Requirements, 
Test Schedule Requirements, Functional Test 
Requirements, Deliverable Requirements, 
Customer Communication Requirements 


Planning 


Generate an 
Evaluation Plan 
TASK #2 



Evaluation Plan, SSR 





Develop a Test Plan 
and Test Procedures 
v TASK #3 






Environmental 

Preparation 


TASK #4 


Redlined Architecture Diagrams, 
Configuration Cut Sheets 



Execution 


Execute a Dry Run 
TASK #5 


Dry Run Report, Daily Debriefs, Redlined Test 
Procedures 



Execute a Formal 
Evaluation 
TASK #6 




Reporting 


Analyze Results and 
create a Test Report 
TASK #7 



Test Report, Lessons Learned 




Figure 12. FEDOS Process. 

FEDOS has 7 tasks to complete a test event; JC3M will consider the Planning Phase 
(Tasks 1 through 3) of FEDOS as an alternative JC3M solution. Each task (identified in 
a block) has outputs (listed in an oval). Each task proceeds when a control gate 
(identified by a numbered circle) is completed. (See Figure 13 for a detailed example of 
FEDOS Task 1.) 
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FEDOS phases were executed in a serial manner, requiring explicit 
approval from all stakeholders before commencement of subsequent tasks. The serial 
nature of FEDOS can be seen in Figure 13, which illustrates only Task 1 of the FEDOS 
process. 



Figure 13. FEDOS Task 1 Elicit Test Requirements. 

This IDEFO illustration of the FEDOS “Elicit Test Requirements” task portrays the 
subtasks (“Elicit Flardware and Software Version Requirements”) used to generate the 
outputs identified in Figure 12. Each process contained exit criteria used as a control 
gate; all stakeholders had to agree on the outputs of a process before the next process 
started. IDEFO illustrations of each alternative are provided at Appendix F. 

b. Marine Air Ground Task Force Command, Control, 
Communications, Computers, and Intelligence Capability 
Certification Test 

MC3T, a system for defining, documenting, and evaluating the 
performance of a MAGTF C4I SoS [Finn, 2007], was identified as the second known 
alternative. MC3T was implemented as a replacement for FEDOS at MCTSSA. 

MCTSSA was charged by MARCORSYSCOM with developing, 
planning, executing, and reporting on a proof of concept event to be conducted in 
October 2007. Three members of this Capstone project team have been actively engaged 

in the development and implementation of MC3T, and will continue to support MC3T as 
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planning, execution, and reporting are conducted. Other MC3T participants include U.S. 
Navy Space and Naval Warfare Center (SPAWAR) Systems Center, Charleston, S.C. and 
U.S. Marine Corps Combat Development Command (MCCDC). 

The first MC3T event will not develop C4I SoS measures of performance 
for consideration, due to time constraints, and will instead be used to prove the validity of 
MC3T processes. During and after the first iteration of MC3T, stakeholders will review 
results to determine if this system will be continued, modified, or replaced. Because this 
review period will occur shortly after the delivery of this JC3M final report, MC3T 
stakeholders are interested in reviewing JC3M results for possible inclusion in future 
MC3T events. 

MC3T has been developed by an Integrated Product Team (IPT) made up 
of stakeholders from MCTSSA as well as representatives of the MARCORSYSCOM 
Product Groups. Product Group representatives defined a "Capabilities Package" 
complete with System requirements, and DoD Architecture Framework Operational 
views and System views that depicted the systems under their cognizance. MCTSSA 
analyzed the Capabilities Package and produced a Consolidated Requirements 
Assessment (CRA). The CRA was an agreement between the stakeholders on what 
needed to be tested, what the required resources were, what the Information Assurance 
compliance requirements were, and what the draft test dates were. Once the CRA was 
approved, MCTSSA produced a Technical Proposal. 

The Technical Proposal defined the technical solution the IPT proposed to 
meet the requirements in the CRA in: staffing, C4I systems architecture design, 
monitoring network architecture design, test cases, data capture and analysis plan, 
information assurance plan, and risk assessment. 

Once the Technical Proposal was confirmed, it became the Tec hn ical 
Solution. The Technical Solution made up approximately 90% of the Test Plan and 
included detailed test procedures and additional reference documentation. 
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c. Joint Test and Evaluation Methodology Capability Test 

Methodology 

The JTEM CTM was chosen as the third alternative. The purpose of 

JTEM is to 

...develop, test, and evaluate M&P [Methods and Processes] for defining 
and using a distributed LVC [Live, Virtual, and Constructive] joint test 
environment to evaluate system performance and joint mission 
effectiveness. JTEM will focus on developing and enhancing M&P for 
designing and executing tests of SoS... [JTEM Rock Drill Event Final 
Report, 2007: i]. 

The five phases of the JTEM CTM are Characterize Test; Plan Test; 
Implement Live, Virtual, Constructive Distributed Environment; and Execute Test 
[JTEM CTM M&P Model Description, 2007: 4-5], Phases 1 and 2 (Characterize Test 
and Plan Test) are considered as a combined alternative JC3M solution and are described 
below. 


CTM.l Characterize Test 

“Initial planning negotiations concerning test program concepts and test 
capabilities are conducted during the Characterize Test phase. Program characterization 
processes include developing the joint operational context for testing, developing the test 
concept, and developing the test’s evaluation strategy. Test capability characterization 
processes include technical assessment producing an initial LVC distributed environment 
test design and programmatic assessment involving high-level test scheduling and test 
resource estimates. Information product inputs to Characterize Test relevant to 
developing joint operational context for testing include the critical CDD, as well as the 
Initial Capabilities Document (ICD), JOpsC Family products, the Universal Joint Task 
List (UJTL), Defense Planning Scenarios (DPS), Combatant Command (COCOM) 
operation plans/orders (OPLAN/OPORD), and the Test and Evaluation Master Plan 
(TEMP). Characterize Test product inputs relevant to test concept and evaluation 
development include an approved AoA plan, test and evaluation strategy, TEMP, and 
operational test plan. The Characterize Test phase produces a test program introduction 
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document (PID) and statement of capabilities (SOC)” [ITEM CTM M&P Model 
Description, 2007: 4], 

CTM.2 Plan Test 

“During the Plan Test phase, test concepts ... are further developed into a 
test plan. Test Planning processes include developing the test design, performing ... 
environment analysis, and coordinating test support, and the synthesis of these processes 
by developing the overall test plan. Developing the test design involves producing test 
vignettes and a data management and analysis plan (DMAP). Performing ... 
environment analysis produces a ... Functional Description. Also, test support 
coordination produces test support plan products. The Plan Test process then produces 
an overall Test Plan, incorporating all identified products from this phase. 

“This Level I division into separate process phases is somewhat of an 
abstraction and a simplification of what occurs in a real test cycle. Many of the Level I 
phases and lower level processes are iterative in nature. Many processes are performed 
in parallel, and the results of these processes are fed back into the iterative work of other 
processes” [JTEM CTM M&P Model Description, 2007: 4], 

The JTEM CTM was tested in a limited scope during 2007, but the 
development of the CTM will not be complete until 2009. The CTM will not be executed 
during the conduct of the JC3M project. The team considered phases 1 and 2 as an 
alternative JC3M solution. All five phases of the JTEM CTM are illustrated in Figure 14. 
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Figure 14. JTEM CTM Phases 

JTEM CTM processes are pictured; JC3M will consider the first two phases 
(Characterize and Plan) as an alternative solution. Dark Blue boxes are CTM phases; 

Green, round edge boxes are inputs and outputs; Execute Test is identified in yellow as a 
simulated event performed in JTEM CTM development. CTM 1 and CTM 2 are 
considered as a JC3M alternative solution [From JTEM CTM M&P Model Description, 

2007: 4], 

2. Generating New Alternatives 

Systems Engineering practice and critical thinking [Paul et al., 2006: 23] call for 
exploration of a full range of options in order to provide good solutions. Alternative 
solutions must be evaluated through a fair minded [Paul et al., 2006: 6] and logical 


36 





























































































































process, using quantitative and qualitative methods to determine the optimum solution for 
the JC3M system. 

The known alternatives discussed above represent the current state of C4I SoS 
evaluation systems, and reflect the resource, risk, and organizational constraints in effect 
at the creation of these systems. The JC3M team generated new alternatives to provide a 
fresh perspective on the problem, in an attempt to generate a better range of solutions. 

The team determined that two new alternatives, in addition to the three known 
alternatives, would provide stakeholders a meaningful range of potential solutions to 
consider. Limiting the alternatives to five would also limit the modeling, simulation, cost 
estimation, and analysis tasks to a scope that could be managed over the duration of the 
project. 

To produce the new alternatives the team utilized a morphological box [Zwicky 
and Wilson, 1967: 9] as a tool to both guide the process and record the results. 

3. Morphological Box Process 

The morphological box (“Zwicky Box”) process began by defining the parameters 
of importance to the problem. For the problem under consideration, the JC3M functional 
hierarchy was used to define the critical parameters to be investigated. These top level 
functions were Define the Problem, ID Systems Under Test (Components of the SoS), 
Define Criteria, and Ensure Readiness. 

A matrix, containing all the potential solutions of the problem, was created. The 
team used brainstorming to generate potential solutions for each separate JC3M system 
function. The brainstorming process consisted of proposing potential solutions for each 
separate JC3M function without consideration of cost, for feasibility, accuracy, speed, or 
any other factor. Ideas were not evaluated, but recorded in a non-graded manner under 
the respective functional column of the alternative matrix. This process was repeated for 
each JC3M system function until the team could not identify or generate any additional 
concepts. 
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All of the generated potential functional solutions were evaluated, and after 
clearly redundant entries were eliminated, the remaining potential function solutions were 


as depicted in Table 2. 


Define the Problem 

ID Systems Under Test 

Define Criteria 

Ensure 

Evaluation 

Readiness 

Have Subject Matter 
Experts do it 

Engineering Document review 

Experience from 
earlier test 

Test Manager 
Review 

Get from CDD 

Depart of Defence Architecture 
Framework (DoDAF) 
Document Review 

Ask JITC 

Program Manager 
(PM) Review 

Conduct survey 

Previous Systems Under Test 

Ask users 

Cost Driven 

Test agency defines 

Ask JITC 

"Trouble Calls" 
recorded on issues 

All Stakeholders 
Review 

JITC defines 

ORD/CDD review 

Build from SoS 
Capabilities 

Planning is finished 

Stakeholders define 

Field Concept of Operations 

Look at Peers (Joint, 
Service) 

Schedule Driven 

Acquisition manager 
defines 

What PM requested 

What PM asked for 

Subject Matter 
Expert Review 

JTEM defines 

Changes to SoS from last test 

Changed capability 
or function in SoS 

Dry Run Test (and 
see if it works) 

Ad hoc definition 

Functional Breakdown 

Work Breakdown 
Structure 

Modeling of test 
plan 

Last definition used 
in previous test 

Bottom up review 

Hardware or 
Software Upgrade 

Run-Test-Run 

Users define 

Test everything 

Ask SPAWAR 

JITC Definition 

User requirements 

Whatever changed 

Test everything 

SAR Review 

What is changed in 
SoS 

Test agency determines 

"Just Do It" (no 
specific guidance) 

ORD/CDD 
Crosswalk , 

Current issues seen 
by SoS users 

System Anomaly Report review 
(SAR) 

ORD/CDD Review 

Change in Test Plan 
from previous event 

(SAR) Generation 

SAR System ID 

Trouble calls from 
the field 


Don't Define It 


Stakeholder Review 



Table 2. JC3M Morphological Box. 

Potential solutions identified for each JC3M sub-function, as generated by Zwicky Box 
process. Top Row is the list of functions, and each cell below it represents a proposed 
solution to that issue, in isolation. 
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Once the team had identified a number of potential solutions for each function, 
the separate potential solutions had to be assembled into a cohesive package for 
consideration as an alternative. One goal of this stage was to identify alternatives that are 
distinctly different. Another goal of this stage was to identify those approaches which are 
attractive, viable, or possible, and to eliminate the approaches which are not attractive, 
viable, or possible. 

The team considered each potential solution to a function separately, and then 
compared the list of potential solutions for the next function in series, looking for 
similarities in theme. As an example, for the “Define Problem” function, one proposed 
approach was to consider “What is changed in SoS.” In order to perform the “ID 
Systems under test” function, one proposed approach was to consider “Whatever 
changed.” For the “Define Criteria” function, one proposed approach was to consider 
“Changed capability or function in the SoS.” For the “Ensure Evaluation Readiness” 
function, one approach was to consider “Change in Test Plan from previous event.” All 
these approaches had the common theme of reviewing only what had changed from a 
previous evaluation, so these were assembled into an alternative. The name for this 
alternative was assigned after members of the team proposed that the underlying theory 
was to consider “what has changed.” 

After review of all potential solutions, nine different alternatives were created by 
linking themes. Table 3 below shows the nine alternatives, named for the theme 
underlying the individual approaches. A detailed description of the proposed alternatives 
is included in Appendix D. 
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Alternative 

Name 

Define Problem 

ID Systems Under 
Test 

Define Criteria 

Ensure 

Evaluation 

Readiness 

Exhaustive 

Don’t Define It 

Test everything 

’’Just Do It” (no 

specific 

guidance) 

Run-Test-Run 

User Defines 

Stakeholders-defme 

What PM requested 

Ask users 

All 

Stakeholders 

Review 

Do No Harm 

JITC defines 

Ask JITC 

Ask JITC 

JITC Definition 

System 

Anomaly 

Report 

SAR Generation 

SAR System ID 

Trouble calls 
from the field 

SAR Review 

Deliberate 

Method 

Stakeholders define 

Functional Breakdown 

Stakeholder 

Review 

Modeling of 
test plan 

Capabilities 

Documentation 

Get from CDD 

ORD/CDD review 

ORD/CDD 

Review 

ORD/CDD 

Crosswalk 

Program 

Manager 

Direction 

Acquisition manager 
defines 

What PM requested 

What PM asked 
for 

PM Review 

Test Agency 
Direction 

Test agency defines 

Test agency 
determines 

Experience from 
earlier test 

Test Manager 
Review 

Change Driven 

What is changed in 

SoS 

Whatever changed 

Changed 
capability or 
function in SoS 

Change in Test 
Plan from 
previous event 


Table 3. JC3M Alternatives. 

Initial JC3M potential solutions, based on JC3M sub-functions, and generated by Zwicky 
box process. Proposed alternatives are composed of like approaches to sub-function 
challenges. 

B. FEASIBILITY SCREENING 

After alternative generation, the JC3M project team reviewed the group of 
potential alternatives to eliminate those which were clearly infeasible, in order to reduce 
wasted effort. The team also reviewed the group of potential alternatives to eliminate 
those which were very similar, in order to present stakeholders with feasible alternatives 
that were sufficiently different so as to provide a variety of approaches. 

After reviewing the nine alternatives from the standpoint of feasibility and 
similarity, the User Defines alternative was eliminated from further consideration 
because it was very similar to the Program Manager Defines alternative. 

The Do No Harm alternative was eliminated because the team planned to consider 

the performance of baseline processes from other organizations (JTEM, MC3T, 
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MCTSSA) from the start, and a primary goal of this alternative generation process is to 
consider two entirely new alternatives, rather than exclusively baseline processes. 

The Capabilities Documentation alternative was eliminated because it is similar to 
the SAR alternative in that both alternatives rely on outside documentation for 
identification of threshold values. 

The revised list of alternatives was left as Ad-Hoc, SAR, Deliberate Method, 
Program Manager Direction, and Change Driven. The team completed this process with 
a smaller set of feasible and sufficiently varied alternatives which could be analyzed over 
the course of the project. In this manner, feasibility screening reduced the nine 
alternatives seen in Table 3 to five, as seen in Table 4. A detailed description of the 
feasibility screening process is included at Appendix D 

Because the FEDOS alternative represented the baseline for C4I SoS evaluation 
performance, each alternative was compared to the baseline for each EM. The team 
reviewed the known performance of the baseline against the projected performance of the 
alternative processes. The scoring criteria were as follows: (+) the alternative was 
expected to perform better than the baseline, (-) the alternative was expected to perform 
worse than the baseline, and (=) the alternative was expected to perform the same as the 
baseline. After evaluating each alternative, the (+) values were summed and the (-) 
values were subtracted, to calculate a final score. Deliberate Method and Change Driven 
alternatives were ranked highest, as displayed in Table 4. 
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Ad- 

Hoc 

SAR 

Deliberate 

Method 

PM 

Direction 

Change 

Driven 

Evaluation Measures 






Percentage of traceable thresholds 

= 

+ 

+ 

+ 

+ 

Percentage of Capabilities Identified 

= 

+ 

+ 

+ 

= 

Time to Plan Evaluation 

+ 

= 

= 

= 

+ 

Number of Traceable Thresholds 
Identified 

= 

+ 

+ 

+ 

+ 

Percentage of Shortfalls Identified 

= 

= 

+ 

= 

= 

Quality of Planning Outputs 

= 

= 

+ 

= 

= 

Number of C4I Systems Under Test 

= 

= 

= 

= 

= 

Percentage of Interfaces Identified 

= 

= 

+ 

= 

+ 

Total + 

1 

3 

6 

3 

4 

Total - 

0 

0 

0 

0 

0 

Overall Total 

1 

3 

6 

3 

4 


Table 4. Reevaluated JC3M System Alternatives Datum Design 
Comparison Matrix. 

This table illustrates the results of the screening of alternatives. The expected 
performance, by function, of each of the potential alternatives solutions was compared to 
the performance of the FEDOS process. Where the potential solution was projected to be 
superior, a “+” is recorded; where inferior, a is recorded. Comparable performance is 
recorded with “=”, and the overall comparisons are summed. “Deliberate Methods” and 
“Change Driven” had the highest scores of the potential alternatives. 

1. Alternative Refinement 

After identifying the most promising new alternatives for consideration as JC3M 
solutions, the team began detailed design of these alternatives, prior to developing 
simulations to evaluate the performance of each alternative. 

In referring back to the original problem statement, the team determined that 
threshold performance criteria must be traceable back to doctrinally correct sources, or 
else they would be perceived as based on personal opinion, rather than Service or DoD 
requirements. In order to provide the most accurate, doctrinally sound, and traceable 
performance measures, the team searched for doctrine guidance, and found it in the form 
of the UJTL. The UJTL is a “.. .comprehensive integrated menu of functional tasks, 
conditions, measures, and criteria supporting all levels of the Department of Defense...” 
[CJCSM 3500.04D, 2005: A-l], 

The UJTL provides a description of what combatants, combat support agencies, 

and DoD activities must perform. More importantly to JC3M, the UJTL provides a list of 
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performance measures and criteria for use in an evaluation. There are also Service- 
specific subsets: the Army Universal Task List and the Universal Naval Task List (used 
by the U.S. Navy, U.S. Coast Guard, and U.S. Marine Corps). 

After some investigation, the team found that doctrinal references provide 
measures (speed, accuracy, duration, % of enemy forces destroyed, etc.) but not the 
threshold values. As an example, OP 3.2.6, “Provide Firepower in Support of 
Operational Maneuver” is a task in the UJTL, and measures include both time to engage 
and accuracy of engagement [CJCSM 3500.04D, 2005: B-B-2], 

In the accuracy measure, percent of enemy forces destroyed, delayed, disrupted, 
or degraded is evaluated [CJCSM 3500.04D, 2005; B-B-2], The UJTL does not state 
what “percent of enemy forces destroyed” is sufficient to constitute success. The 
threshold value is left out by design, in order to allow the commander the flexibility to set 
a threshold in accordance with the mission, the conditions, and the situation. 

Service-level guidance was reviewed as well. Marine Corps Training and 
Readiness Manuals list measures and criteria, but they focus on functions and not 
capabilities. As an example, the [Artillery T&R Manual, 2000: 3-IF-45] describes the 
task of controlling a battery of towed artillery pieces moving in a convoy, and responding 
to an emergency request for artillery support by halting and placing the guns to fire on a 
target. "Lay the Battery for an Emergency Fire Mission (Hip Shoot) while in a Convoy” 
is stipulated as a performance task, but the criteria are not provided. 

The team determined that an attempt to define threshold performance values 
would violate the spirit, if not the letter, of DoD and Service level guidance. Further, if 
the JC3M system were to define threshold performance values, in spite of DoD guidance 
to the contrary, the recommended thresholds would be subjective, and subject to 
immediate challenge by operators, subject matter experts, acquisition managers, and 
other testers. If the results of a test are perceived to be subjective, they are worth very 
little. 

The team also investigated process or system based solutions which could 
generate threshold values, but could not find any such solution. There are tools in 
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development [Lockheed, 2007] that will assess conformance to standards and 
interoperability based only on technical exchange of information. Tools will not compare 
performance for end-to-end operational effectiveness for mission accomplishment, only 
compliance to a technical standard. Interoperability must include “... both the technical 
exchange of information and the end-to-end operational effectiveness of that exchanged 
information as required for mission accomplishment” [CJCSI 3170.OIF, 2007; GL-9]. 

Based on the number of possible conditions, missions, unit types, and components 
of the SoS, the team determined it would be futile to try to create a tool that could 
account for all the combinations 

2. JC3M Changes 

Without doctrinal sources for, or processes to generate, threshold criteria, the 
team determined that there were several potential changes to JC3M. One 
recommendation to stakeholders could be to assemble the SoS as described by 
stakeholders, test against performance measures, then record the performance. This 
would establish a baseline of performance values, so that when SoS components were 
changed in the future, testers could ascribe the changed results to changed components. 
("With Box X, missile speed was 620 mph; with Box Y installed in place of Box X, and 
everything else held constant, missile speed was 670 mph.") The before and after 
comparison would be very objective, and would allow stakeholders, not testers, to 
determine if the performance was acceptable. This is the methodology used at Naval 
Surface Warfare Center (NSWC) Corona Division, and will be used by MC3T at 
MCTSSA for the proof of concept event in October 2007. 

Another methodology would be to recommend that testers use doctrinal 
references to objectively define only measures, test the SoS to evaluate its performance 
on those measures, and report the results to the stakeholders ("87% of enemy forces were 
destroyed"). This approach depends on the ability to assess combat-focused performance 
on measures that are not typically available in a test organization for space, time, and 
safety constraints. While simulations can be used to accomplish this approach, this adds 
another layer of complexity to the conduct of a C4I SoS evaluation, as well as the 
potential for challenges to the evaluation methodology, based on the implementation of 
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the combat simulations. This approach was considered, but because JC3M is focused on 
objective measures, it was not considered. 

Another methodology would be to recommend further emphasis on the JCIDS, 
with all future acquisitions made on a SoS level, with objective performance criteria at an 
operational level identified in JCIDS documents before acquisition. This is, in the long 
run, the best answer for DoD, but does not address the many legacy C4I systems that are 
assembled, after the fact, into a SoS. This is also in line with the concept of capability 
portfolio management ordered by Deputy Secretary of Defense Gordon England in 2006. 
In a memo defining capability portfolio management roles and responsibilities, Secretary 
England stated the intent is to “... manage groups of like capabilities across the enterprise 
to improve interoperability, minimize capability redundancies and gaps, and maximize 
capability effectiveness” [England, 2006: 3], The memorandum also charged the Joint 
C2 Capability Portfolio manager to create a test case that “.. .delivers integrated joint 
capabilities, which improve interoperability, minimize capability redundancies and gaps, 
and maximize joint operational effectiveness” [England, 2006: B-l], The Joint C2 
Capability Portfolio manager, with three other portfolio managers, worked on the 2008 
DoD Budget request, which started the process of adjusting Service C2 programs 
[Federal Computer Week, 2007]. While an excellent long-term solution, this does not 
address the immediate need for performance measures to use in evaluating legacy C4I 
SoS. 

The team chose to use JC3M to define performance measures only, and use those 
measures to test the C4I SoS. This represents a change to the JC3M effective need, i.e. 
instead of pursuing threshold performance values (“60 mph is the passing criteria”), 
JC3M will define the performance measures to be used in the evaluation of a C4I SoS 
(“measure the speed of the vehicle”). 

The team reviewed the generated alternatives and determined they would have 
been effective when the original goal of JC3M was to provide a method for organizations 
to define threshold performance criteria for use in C4I SoS evaluations. 
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When documenting the processes that comprise each of the alternatives, prior to 
creating models, the modeling and simulation team determined that in defining C4I SoS 
performance measures, both Deliberate and Change-Driven alternatives performed the 
same functions, in the same manner. As originally defined, each model took the same 
external data sources (the UJTL, Systems Engineering Artifacts, and system-level 
documentation) into consideration when defining performance measures. The 
alternatives differed in how they performed the next step, i.e. identifying the threshold 
values which were the goal of the original JC3M solution. Because those threshold 
values were no longer pursued, the final and discriminating phase of each alternative was 
eliminated. With that elimination, the modeling and simulation team determined that PM 
Direction and SAR would perform exactly alike, thus eliminating their value in 
presenting different approaches to the JC3M solution. 

The team generated two new alternatives to meet the revised JC3M requirements, 
utilizing the same functions that drove the first round of alternative generation, again 
using the “Morphological Box Process” described in Chapter III section A.3. 
a. Systems Capabilities Review Alternative 
The Systems Capabilities Review (SCR) alternative is a combination of 
the “Capabilities Documentation” alternative and the “Test Agency Direction” alternative 
outlined in Table 3. SCR is composed of a group of stakeholders: C4I system and SoS 
user representatives, test agency representatives, system developers, system designers, 
and program managers. The test agency representative chairs the group, which meets as 
required to support a C4I SoS evaluation, at the Systems Command level. 

Inputs to SCR include source documents such as the Capabilities 
Development Document (CDD), Operational Requirements Document, Test Plans, 
TEMPs, Concept of Operations documents, Joint Integrating Concepts, Joint Operating 
Concepts, and system level metrics. 

The SCR first reviews SoS capabilities specifications, and will examine 
the systems engineering artifacts already created, such as supporting DoD Architecture 
Framework documents and technical performance measures, and creates a list of implied 

and stated SoS capabilities. Next, the SCR reviews system-level documents and creates a 
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system-level capabilities list. Third, the SCR maps system-level capabilities to SoS EM. 
The SCR identifies gaps in the EM list, and creates the balance of the EM necessary to 
evaluate the performance of the C4I SoS. Figure 15 illustrates how SCR performs the 
JC3M subfunction 1.3.2 “Define Measures.” 


SCR Alternative 



Figure 15. List of Sub Tasks Performed by SCR to Define Measures. 

The JC3M Functional Hierarchy calls for function 1.3 “Define Evaluation Criteria.” This 
function is supported by JC3M subfunction 1.3.2 “Define Measures.” The SCR 
alternative, in order to support 1.3.2, performs sub processes. Each of the subprocesses 
may have multiple submissions from C4I SoS evaluation stakeholders. 
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The subtasks performed by SCR as part of Task 1.3.2 “Define Measures” 


are illustrated in more detail in Figure 16. 



Figure 16. SCR Define Measures Subtask Ordering. 

This figure illustrates the subtasks broken down in the order in which they are performed 
in the SCR alternative. Review SoS Capabilities, Review System Level Documents, and 
Define Measurement Conditions and Rules can be worked in parallel. Once these tasks 
are complete the team maps EM SoS capabilities and reviews gap areas. These functions 
are consistent with the more detailed functional analysis performed on the new 
alternatives. “REF” is a CORE artifact, and indicates no additional function is enabled 
when this flow completes. 

b. Functional Capabilities Board Alternative 
The Functional Capabilities Board (FCB) alternative is based on an 
existing group, the JCIDS C2 Functional Capabilities Board, to define the performance 
measures of the C4I SoS. Figure 17 illustrates the tasks of an FCB, which is to perform 
“... organization, analysis, and prioritization of joint warfighting capabilities within an 
assigned functional area” [CJCSI 3170.OIF, 2007; GL-7]. Inputs to the FCB include the 
UJTL and subsets, Concept of Operations (CONOPS) documentation, Program Change 
Request documentation, and system trouble reports. 


Standing FCBs evaluate shortfalls and gaps in current and future 
capabilities as they are identified by studies or assessments, or arise from processes or 
meetings [CJCSI 3137.01C, 2004: E-l], The FCB reports are provided to the Joint 
Capabilities Board (JCB) or Joint Requirements Oversight Council (JROC). The 
additional effort proposed in this alternative represents an increase in the work performed 
by the C2 FCB, but is in the same functional area, and engages in many of the same 
tasks. Unlike the SCR, the FCB meets on an ongoing basis, rather than as required to 
support C4I SoS evaluations. 
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The FCB will first identify the configuration of the SoS by determining 
the component systems. Next, the FCB will identify the SoS capabilities. SoS CONOPS 
are reviewed to determine EM, and finally the FCB will generate the SoS EM list for use 
in C4I SoS evaluations. The FCB, under JCIDS, has a long-term perspective and charter. 
The FCB represents a near-term resource that can provide ongoing analysis, and so 
reduce the lack of C4I SoS performance measures. The FCB appears to explicitly support 
the concept of capability portfolio management, but the JC3M team could not find any 
information to confirm that relationship. 
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Figure 17. JCIDSandFCB. 

FCB is central to JCIDS and provides Service and Functional Area representation as well as analysis of existing capabilities and future 
shortfalls in capabilities. FCB analysis is provided to the Joint Capabilities Board and Joint Requirements Oversight Council [From CJCSI 
3137.01C, 2004]. 
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The external agency relationship between the FCB and C4I test 
organizations, and the list of sub tasks needed to complete the Define Measures task, is 


illustrated in Figure 18. 



Figure 18. List of Sub Tasks needed to Complete Define Measures. 

The FCB Alternative performs the task of defining measures, using a team external to the 
test organization that meets weekly throughout the year. Once the test agency enters their 
planning process for a C4I Evaluation, they elicit input from the FCB team on capability 
measures. These measures are analyzed for feasibility and testability before they are 
included in data collection and analysis. 

Because the FCB is external to the test organization, some analysis of the 
performance measures generated by the FCB will be necessary. Figure 19 illustrates both 
the external nature of performance measure generation, for this alternative, as well as the 
required analysis performed internally to the test organization. 
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Figure 19. FCB provides external input to C4I Test Organization. 

This illustration highlights that the FCB is external to the Test organization, and both 
organizations continuously coordinate their efforts. The FCB meets and generates 
performance measures independent of the test organization’s schedule. From a planning 
perceptive; the test organization need only analyze incoming measures for feasibility and 
testability. 


c. Differences between SCR and FCB 

From the JC3M Functional Hierarchy perspective, SCR and FCB appear 
the same. The difference between these alternatives is in the approach each alternative 
takes to complete process 1.3.2 “Define Measures” in the JC3M Functional Hierarchy, as 
illustrated in Figure 20. 
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SCR incorporates 
all the tasks 
associated with this 
process into the test 
planning process 


FCB utilizes an 
external Team that 
meets year round to 
provide Capability 
Measures to the test 
agency 


Figure 20. SCR and FCB Variance. 

The FCB and SCR alternatives differ in subtasks and their approach to JC3M function 
1.3.2 “Define Measures.” SCR performs this task with internal resources, and in 
response to requests for C4I SoS evaluation. FCB is a resource external to the C4I test 
organization, and performs this task weekly. 

The alternatives differ in the source of personnel required to execute a C4I SoS 
evaluation; whether they have been used in a historical event, have been simulated, or are 
a new proposal; what organization chartered their development and execution; and the 
sources used to define performance measures. Table 5 summarizes the key similarities 
and differences between the alternatives. 
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Personnel 

Resources 

Used for 
Testing 

Service or Joint 

Performance 
Measure Creation 

FEDOS 

Internal 

Yes 

Executed by 
Service test 
organization, 
driven by Service 
stakeholders 

Stakeholder 

agreement 

MC3T 

Internal 

Yes 1 

Service level 
charter for 
certification of 
C4I capabilities 

Doctrine 
Developers and 
C4I system 
stakeholders 

JTEM 

CTM 

Internal 

No 2 

Developed for 
joint use 

Review of 
doctrine, system 
documentation 

FCB 

Internal / 
External 

No 

Proposed for joint 
use 

Review of Doctrine 
and system 
documentation by 
Joint C4I SME 
Panel 

SCR 

Internal 

No 

Proposed for joint 
use 

Doctrine 
Developers and 
C4I system 
stakeholders 


Table 5. Summary Comparison of Alternatives. 

The table summarizes key features of the alternatives under consideration. Resources 
indicate if personnel used in the conduct of a C4I SoS evaluation are internal or external 
to the test organization. Used for Testing indicates if the alternative has been used at a 
test organization. 'MC'3T was projected to execute a proof of concept in October 2007. 

2 JTEM CTM has executed simulations, and was projected to conduct limited-scope tests 
in late 2007 (results not available for review). Origin indicates the organization that 
ordered the development or execution of the alternative. JTEM CTM is a Joint Test and 
Evaluation project chartered by the Director of Operational Test and Evaluation. Source 
of Performance Measures indicates where performance measures are generated. 

C. ALTERNATIVE SCORING CRITERIA 

After the revisions to the JC3M approach, revised EM were required to rank the 
alternatives. The original nine EM created as part of the JC3M value hierarchy, and 
documented in Appendix D, were reviewed, and those specific to threshold values 
(Percentage of Traceable Thresholds, Number of Traceable Thresholds Identified) were 
eliminated. EM that measured the SoS configuration (Percentage of Capabilities 
Identified, Percentage of Shortfalls Identified, Number of C4I SUT, Percentage of 
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Interfaces Identified), rather than measuring the production of performance measures, 
were eliminated. 

Those EM that measured how well the alternative performed (Percentage of 
Traceable Measures, Quality of Planning Outputs) were retained, as well as those EM 
that measured the duration of the alternative process (Days to Plan Evaluation). Labor 
Hours were retained, and though not seen below, were used in the Life Cycle Cost 
Estimate (LCCE) for each alternative. Finally, those EM that measured the flexibility of 
the alternative (Elasticity of Labor, Elasticity of Duration) were also retained. Table 6 
lists the revised EM used to rank the performance of each alternative. 



Percentage 
of Traceable 
Measures 

Days to 
Plan 

Evaluation 

Quality of 
Planning 
Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

Alternatives 

1.3.2 

1.4.3 

1.4.3 

4.1 

4.1 

FEDOS 






MC3T 






ITEM CTM 






FCB 






SCR 







Table 6. JC3M Scoring Matrix. 

EMs used for comparing alternatives measure traceability of outputs, duration, quality of 
outputs, and the response of the alternative to changes in SoS size. Labor hours were later 
recorded, but used in the LCCE, required for the final Cost/Benefit analysis. 
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IV. MODELING AND SIMULATION 


A. JC3M USES OF MODELING AND SIMULATION 

Modeling and Simulation (M&S) was the most important element of the JC3M 
SE process, because it allowed alternative solutions to be compared in an affordable, 
effective, and repeatable manner. If real systems could be utilized in their operating 
environment, comparisons would be of higher fidelity, but this is not often possible. In 
many situations, safety, practicality, or resource constraints make it impossible to utilize 
alternatives in their proposed environment. It is not acceptable, for example, to provoke 
a war to test the effectiveness of a defensive system. It is also not prudent to purchase 
and field a system, without an acceptable estimate of performance, before the start of 
hostilities. The team could not build facilities and implement each alternative process 
for C4I SoS evaluation in order to compare their effectiveness and measure their costs, so 
M&S was used. 

A model is a physical, visual, or mathematical representation of a real system that 
accounts for its known or inferred properties and may be used for further study of its 
characteristics [Forsberg, Mooz, Cotterman, 2000: 18]. The purpose of a mathematical 
model is to represent key characteristics of the real system that we want to study, such as 
operating cost, productivity, or time to market. The purpose of a visual model is to allow 
analysis of key relationships, inputs and outputs, and functional flows of the real system. 
Software tools were used to build both mathematical and visual models of each 
alternative. 

Simulation operates the model with selected inputs, outputs, and environmental 
conditions in order to analyze the performance of the system. Models of each alternative 
were generated, and simulations were executed on the models utilizing different data 
inputs in a systematic and logical manner to gain insight into how each alternative would 
perform. The outputs from the models and simulators were analyzed by the team, and the 
results provided input into the Analysis of Alternatives phase. With the exception of the 
FEDOS alternative, none of the alternatives under consideration have historical 
performance data available for analysis; therefore modeling and simulation was an 
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effective method to analyze the alternatives, predict behavior under varying conditions, 
and compare the performance of the different alternatives. 

Figure 21 identifies two of the modeling and simulations used to generate three of 
the five EM used to compare the performance of the alternatives. POW-ER (Project, 
Organization, and Work for Edge Research) refers to a modeling and simulation tool 
developed by Stanford University to optimize the work flow of an organization. Arena 
refers to the Rockwell Automation tool that can model how a system behaves over time. 
POW-ER, Arena, and CORE (another M&S tool) are described in detail in Section B 
“JC3M Modeling Tools.” In Figure 21, “offline evaluation” refers to the consultation 
with SMEs (described in Appendix C) that was used to generate the remaining EM. 



Figure 21. M&S Contribution to JC3M Scoring Matrix. 

This figure illustrates the three EM which relied on M&S for data. The “offline 
evaluation” in the figure refers to the SME consultation described in Appendix C. 

M&S enabled the team to predict the behavior of an alternative when input 

parameters changed. The team was able to identify what would happen if the input 
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parameters to a system increased by 10%. Would the labor hours and calendar duration 
increase by the same amount? It was important to be able to answer these questions, 
because they provided insight into the behavior of the alternatives. The effects on 
duration and labor hours related directly to the elasticity of the alternative. 

B. JC3M MODELING TOOLS 

The M&S tools used in the project allowed the team to generate models and run 
simulations to analyze the required EM for each of the JC3M alternatives. 

1. CORE 

CORE is a systems engineering software tool that provided traceability from 
Problem Refinement through AoA. The team used CORE to construct standard 
Integration Definition for Function Modeling (IDEFO) diagrams. The IDEFO diagrams 
provided the reference architecture for analysis of the alternative systems. The IDEFO 
diagrams allowed the team to identify and analyze the functions that are performed by 
JC3M, as well as by each alternative, and to record the mechanisms (means) by which 
they were performed. The graphical representation of the systems provided by CORE 
through the IDEFO permitted the team to conceptualize the functional requirements of the 
systems. IDEFO diagrams helped to facilitate understanding, communication, consensus 
and validation of the systems under study [Federal Information Processing Standards 
Publication, 2007: 183], Samples of the CORE IDEFO analyses are provided in 
Appendix F. 

2. POW-ER 

POW-ER (Project, Organization, and Work for Edge Research) is a project 
organization modeling and simulation tool that integrates organizational and process 
views. POW-ER is developed as an outgrowth of the Virtual Design Team (VDT) 
computational modeling research at Stanford University. POW-ER addresses the 
organizational elements that impact the ability to work effectively, including: policies 
and structures (culture, communication, decisions, meetings); staffing, hiring, and 
training needs for workforce plans. Using POW-ER, the team modeled the 
organizational structure, the relationship between individuals within those organizations, 
and individual task allocations. 
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The Days to Plan Evaluation EM, identified in the M&S Contribution to JC3M 
Scoring Matrix (see Figure 21), is directly related to the management and organization 
aspect of the JC3M system. POW-ER allowed experimentation with “What-If ’ scenarios 
and provided insight into the level of rework, backlog, and risk associated with an 
alternative. POW-ER’s ability to predict and analyze backlogs proved to be quite useful 
in the design and troubleshooting of alternative models, because it allowed the team to 
identify backlogs in the workflow of models. The analysis of backlogs in the workflow 
enabled the team to identify the optimized arrangement of tasks and personnel for both 
the SCR and FCB alternatives. 

Each POW-ER model was generated by adding the tasks and personnel identified 
for an alternative from their respective data sets (details provided later in this chapter) 
Tasking flow was modeled based on the defined relationship from the data sets (i.e., 
parallel tasks were modeled to occur simultaneously and serial tasks were modeled to 
occur based on dependencies). Next, personnel that perform similar tasks were assigned 
into a team; those teams were then linked to a respective task. 

Both parallel and serial task relationships can be seen in Figure 22, which depicts 
examples of each of the top level functions (Elicit Requirements, Generate Evaluation 
Plan, Develop Test Plans and Procedures) of the FEDOS process. 
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Figure 22. POW-ER Baseline Model. 

Parallel and Serial tasks are modeled in POW-ER, based on historical performance 

(FEDOS), stakeholder input (MC3T, JTEM CTM) or design of the alternative (FCB, 

SCR). Models contain both tasks and personnel assigned to the tasks. 

Within POW-ER, the Effort attribute for each task was set by assigning it to the 
duration in days identified in an alternative’s data set. For the Team attribute, the Full 
Time Equivalent (FTE) attribute was set equal to the total hours an alternative team spent 
on each task divided by 40 hours (one work week). 

It is important to mention that the POW-ER simulation tool calculates a task’s 
duration in one of two ways: simulated duration or Critical Path Method (CPM). 
Simulated duration factors in the “hidden works” that the CPM does not. The “hidden 
works” associates an amount of rework and delays into each task. The simulated 
duration was selected for Days to Plan Evaluation EM, because it is fair to assume that 


61 










































































these hidden works are always part of each task. The accuracy of the POW-ER model 
was validated against historical performance data (details provided later in this chapter). 

3. ARENA 

Arena provided a numerical evaluation of a system by imitating the system’s 
operations or characteristics over time. Arena allowed the team to conduct numerical 
experiments in order to predict the behavior of an alternative given a set of conditions. 
The JC3M Scoring Matrix identifies two EM (see Figure 21) that required evaluation of 
the changes in output as a function of the changes in inputs: Elasticity of Labor and 
Elasticity of Duration. Arena allowed the team to run simulations on the alternative 
models with varying set of inputs; Arena displayed the output changes that corresponded 
to each of the varying inputs. 

Figure 23 shows results of a simulation run in Arena that predicts the duration and 
labor hours require for planning a SoS evaluation. As noted, controls can be varied. 


Input Variables can be 
altered to collected the 
simulated responses 



Duration 


Figure 23. Arena Simulation Run. 

Simulation allowed the team to change input parameters to reflect the concerns of 
stakeholders. Outputs were measured, in response to input changes, to analyze the 
performance of the alternatives in changing situations. 
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The team created a baseline C4I SoS to use in comparing the performance of each 
alternative in Arena (See Section C “Select a Baseline” and Section D.l “Baseline C4I 
Architecture” for details of the baseline C4I SoS). After the models of alternatives were 
created (see Section C.7 “Build POW-ER, Arena, and CORE Model for each 
Alternative”), Arena was used to calculate the duration of each task. Systematic 
variations to the inputs (number of individual systems and number of capabilities) were 
made to analyze how changes in the size of the C4I SoS affected the EM of interest. 

From the varied inputs, and resulting outputs, the elasticity EM needed for the AoA was 
calculated. 

All Arena models are based on data sets containing process tasks, job billets, and 
hours worked. For process tasks that contained sub-processes, the team created sub 
models as illustrated in Figure 24. 
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Baseline Planning Process Model 
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O utput needs to be rerthh 5*fo of: 6,4S2 T PtafTrne ri Man liars 



Figure 24. Example of Arena Sub-Models and Sub-Processes. 

Each alternative’s Arena model had sub-models that contained a series of process delays 
that affected Duration and Total Labor hours. The number of sub-models and associated 
tasks depended on the design of the alternative. 

The sub models used in Arena contain process delays, as appropriate, which were 
used to represent average labor hours per variable instance. Figure 25 illustrates 
variables, units, and the mathematical expression used in calculating an example process 
delay. 
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“OldCaps”, “NewCaps”, and “SysNum” are variables 
that are manipulated by the Process Analyzer to 
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values represent average hours spent on each 
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Figure 25. Example of a Process Delay Setting. 

Each sub model contained a series of process delays used to represent labor hours per 
variable instance. The variables identified ( OldCaps, NewCaps, and SysNum) were called 
and varied in a centralized process analyzer which enabled the modeling team to run 
multiple variations of each alternative. 

Each alternative was modeled in Arena, and imported into the Arena Process 
Analyzer for efficient simulation of the alternative using varied inputs. The team utilized 
the control feature to centrally control and vary input variables to study the associated 
variability of outputs. Each model used three critical input parameters to determine the 
size and the scale of the test event. These parameters are: 

■ Number of Systems (SysNum} - This parameter did not represent the 
number of instances of a system (i.e., 14 AFATDS terminals), but 
represented the number of types of systems under test. If the SoS 
contained PLI systems, communication systems, and fire control systems, 
SysNum equaled 3. 

■ Number of New Capabilities f NewCaps ) - This parameter represented the 
number of SoS capabilities under evaluation. If the C4I test organization 
had little to no experience with this capability, it was considered a new 
capability. 
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■ Number of Old Capabilities (OldCaps) - This parameter also represented 
the number of SoS capabilities under evaluation. If the C4I test 
organization was familiar with and had tested the capability, it was 
considered an Old Capability. This parameter addressed the concept of 
efficiency associated with experience. 

Figure 26 shows a view of the Process Analyzer tool, and illustrates the multiple 
versions of each alternative and the varying magnitudes of NewCaps, OldCaps, and 
SysNum. 
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Figure 26. Screen Shot from the Arena Process Analyzer. 

Arena Process Analyzer is used to centrally vary input variables for each alternative 
model in order to examine the effects on the performance of the alternative. In this 
illustration, the alternatives are identified in the left column, with sizes varied by 25% 
steps. This data is used, in turn, to conduct analysis of the performance of each 
alternative. 
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C. MODELING AND SIMULATION APPROACH 

The JC3M team implemented an eight step M&S approach. The details of this 
approach are provided for clarification. 

1. Select a Baseline System 

A baseline of system performance was selected that could be used to validate the 
models created by the team. The team chose the FEDOS alternative for designing and 
validating performance models of all alternatives. Use of FEDOS allowed the team to 
compare the performance from M&S tools to the historical data documenting the actual 
performance of FEDOS, thus validating the accurate performance of the model. 

Inputs such as number of systems, capabilities, and tasks were gathered from the 
historical record, models were designed, and simulation outputs were generated. Outputs 
were compared to actual results, and the model was systematically adjusted until it 
produced results that were close to actual labor hours and duration (Days to Plan 
evaluation) of the historical event. The labor hour calculations were validated to within 
±1% of historical values because this data was used was used to calculate the LCCE of 
the alternatives. Duration was validated to within ±4% of historical values because this 
EM only contributed 5.8% to the overall performance score (see Chapter VI, Section E 
“Analytical Hierarchy Process”) of the alternative. This provided the team confidence 
that the models built in Arena and POW-ER accurately emulated the performance of the 
FEDOS alternative, and could be used to accurately simulate the performance of other 
alternatives. 

As noted earlier, each of the five alternatives focused on the planning of a C4I 
SoS evaluation. Figure 27 illustrates the Planning function of JC3M. 
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Figure 27. JC3M Planning Functions. 

The models of alternatives were validated by comparison to FEDOS historical 
performance. After adjustment of the models to produce results within 1% of historical 
FEDOS performance, each of the alternatives was modeled in turn. This illustrates the 
entire planning functional hierarchy of JC3M; Figure 11 only illustrates the critical EM. 

2. Select a Candidate C4I Test Architecture as a Baseline SoS 

The JC3M team selected the C4I SoS architecture tested in 2005, using the 
FEDOS process, as stated above, because historical performance data existed which 
allowed consistent comparison of the models to actual results. The architecture that was 
the subject of the FEDOS 2005 Event was composed of nineteen different types of 
systems (described in Section D “Baseline Model Development” of this chapter). The test 
evaluated fourteen SoS capabilities; ten were previously tested (“old”) and four were new 
to the test team (“new”). Each of the alternatives was modeled with this architecture as 
the C4I SoS under evaluation. 

This consistency of C4I SoS under evaluation is important because the FEDOS 
2005 was a real SoS evaluation event that could be modeled and simulated. The results 
of simulation could be compared to historical performance data. Further, establishment 
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of a baseline C4I SoS allowed for an “apples to apples” comparison of the performance 
of each alternative, as they were used to plan the evaluation of the same C4I SoS. 

3. Assemble a Data Set for the FEDOS System 

A data set is the compilation of data (duration of tasks, resources, sequence of 
events, labor hours) used to populate the model. In this case, historical documents that 
recorded the performance of FEDOS 2005 were used as the sources of data that 
documented the configuration of the baseline system. Three historical documents were 
used to build the baseline data set. 

The FEDOS 2005 Technical Support Plan (TSP) was a Microsoft Project® plan 
that specified the tasks performed and the order in which they were performed; identified 
the resources used for each task; and identified serial task dependencies and parallel 
tasks [Manning (b), 2005]. The Labor Hours report [Manning (a), 2005] recorded the 
hours worked by each employee for each task. The Test Report [Marine Corps Tactical 
Systems Support Activity, 2005] provided the evaluation of the conduct of FEDOS in 
2005. Figure 28 provides an illustration of the process of conversion of historical data 
into the baseline data set. 



Figure 28. JC3M Baseline Data Set Inputs and Outputs. 

The baseline data set was built from three historical records, and was used to populate the 
models of each alternative in turn. This allowed validation of the models against 
historical data, as well as evaluation of the performance of each alternative against the 
same size and scope C4I SoS. 
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Data from the TSP, Labor Hours Report, and Test Report were summarized in a 
spreadsheet that consisted of the following list of data for use by the models of each 
alternative: 

■ a comprehensive list of planning tasks used for test planning, 

■ labor hours for each test team member (per task), 

■ calendar duration for the planning process, 

■ inputs and outputs for each task, 

■ each test team member’s cost per hour, 

■ test team position such as Test Lead, or Technical Writer, and 

■ organizational attributes such as duration of work day, duration of work 
week. 

4. Model, Simulate, and Validate the Baseline Processes 

In this step, a model of one alternative, (FEDOS) was created from the baseline 
data set in POW-ER, ARENA and in CORE. (A model of each of the other four 
alternatives is created in step 7.) Figure 29 illustrates the data set inputs to each modeling 
tool and the outputs from the simulations. 
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Input to Models 


Output 


Baseline Data Set 


■List of Planning Tasks 


■Labor Hours per Task- 


■ Hours Worked per Team Member* 


■Task Priority- 


■Position Attributes- 


Labor Hours per Task^^— 
i Hours Worked per Team Member* 
List of Planning Tasks 


CORE 


POW-ER 


ARENA 


■IDEFO Diagrams- 


■Days to Plan Evaluation- 


■Elasticity of Duration- 
—Elasticity of Labor* 


Figure 29. Input and Output of Baseline M&S. 


CORE provided IDEFO Diagrams for functional analysis (see Appendix F). POW-ER 
provided Days to Plan Evaluation. Arena provided labor hours (used for Life Cycle Cost 
Estimate) and Elasticity of Labor and Duration for each alternative. 


5. Map the Alternatives to the Baseline Processes 

Each task in each of the alternative tasks was mapped (where possible) to the 
baseline processes. This allowed the test team to use known labor hours and cost from the 
baseline to complete an estimation of cost and duration for alternative. Figure 30 shows 
some of the tasks from the baseline and their mapping to the JC3M functional hierarchy. 
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The JC3M Modeling Team 
Mapped Baseline Processes 
to the JC3M Functional 
Hierarchy in order to 
associate real world data to 
the JC3M Functions 


1 

r 

2 

Baseline Planning Process / 

\ JC3M Functional Hierarchy 

3 

Task Niinie I 

Task Name 

1 

1.1 Elicit Hardware and Software Version Requirements 


5 

1.1.1 Elicit HW/SW versions from the Customer (Create and Circulate Questionaire and attend weekly WG meetings) — 

j> 1.2.1 ID SUT 

6 

1.1.2 Elicit HW/SW Delivery Dates (Fill out the Questionaire and attend weekly WG meetings) - 

t 1.2.1 ID SUT 

7 

1.1.3 Create System s List (Weekly updates to this list and rework as changes come in) 

► 1.2.1 ID SUT, 111 ID Short Falls 

8 

1.1.4 Circulate the System's List for Customer Review (Attend weekly WG meetings and send out a weekly update) - 

► 1.2.1 ID SUT 

9 

1.1.5 Get Customer concurrence for System's List at Control Gate 1 - 

► 11.2 Conduct Review 




V 


Figure 30. Mapping of Baseline Tasks to JC3M. 

This figure illustrates the traceability of tasks to the JC3M Functional Flierarchy. In the 
figure, processes from the baseline are mapped to JC3M tasks. Because many tasks were 
common from alternative to alternative, baseline tasks could be mapped to each 
alternative model, ensuring consistency of comparisons, traceability of inputs and 
outputs, and reducing variability. 

6. Create Data Sets for Each JC3M Alternative 

After the mapping was complete, each alternative’s data sets were assembled 
from the baseline data. Each alternative’s tasks and sub-tasks were mapped as close as 
possible to baseline data. The duration and labor hours were assigned based on this 
actual data. If an alternative had tasks that did not map to the baseline, MCTSSA SME 
input was used to estimate and validate duration and labor hours. 

7. Build POW-ER, ARENA, and CORE Model for Each Alternative 

In step 4 a POW-ER, Arena, and CORE model was developed for the Baseline. In 
step 7 a model of each of the remaining four alternatives was developed in each of the 
three modeling tools. Figure 31 depicts a high level view of this step. 


72 




























Input to Models 


Output 



Figure 31. Input and Output of Alternative M&S. 


This is a diagram of Alternative inputs to CORE, POWER, and ARENA, and their 
associated simulation results. 


8. Run the Models and Capture Simulation Results 

Each POW-ER model was run to generate the Days to plan an evaluation EM. 
For each ARENA model, simulations were run to generate Elasticity of Labor and 
Elasticity of Duration EM. Each CORE model generated IDEFO diagrams. 

D. BASELINE MODEL DEVELOPMENT 

Each JC3M alternative system represented test planning processes. In order to 
compare each alternative’s performance, each model’s parameters must be consistent in 
terms of number of systems under test and number of capabilities under test. 

1. Baseline C4I Architecture 

The team elected to model the C4I SoS that was under test during the FEDOS 
2005 test event as the baseline system. This C4I SoS was selected because it had 
had complete employee time sheets, a project schedule, and the TSP data that recorded 
the cost associated with the planning process. This data gave the team valuable insight 
actual labor hours, duration, and cost to plan a SoS test event. The architecture for the 
FEDOS 2005 event SoS consisted of ninety-one physical systems comprised of nineteen 
types under test, as listed in Table 7. 
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Systems 

Software / Firmware Version 

Hardware Version 

AFATDS 

6.4.0.0UMR92Z 

CCU2/Tadpole 

CLC2S 

2.0.5 

T-40 

CONDOR 

3.6.5.0.0.9 

N/A 

DDACT 

4.0.0.0.12D 

RPDA 5500 

ENM 

4.4 

CF28 

EPLRS 

11.4 

RT-1720 

GCCS-J 

4.0.1.0 

Netra 240 

GCCS-J Client 

4.0.1.0 

T-40 

IOS VI 

4.0.1.1.01 

Netra 240 

IOS V2 (Apps) 

3.7.0.1.01 

Netra 240 

IOS V2 (SDS) 

3.7.0.1.01 

Netra 1125 

IOW VI 

4.0.1.1.07 

T-40 

IOW V2 

3.6.6.1.01 

T-40 

MDACT 

3.6.5.0.12 Final 

RHC31A 

PFED 

1.0 

RPDA 5500 

SINCGARS 

N/A 

RT-1523A, RT-1523B, RT-1523E 

SISTIM 

6.4 

CCU2 

TBMCS Lite 

1.1 

Sunfire 

TBMCS Remote 

1.1 

T-40 


Table 7. Systems Under Test during FEDOS 2005. 

The architecture for the FEDOS ’05 SoS evaluation consisted of nineteen types of C4I 
systems. Ninety-one physical systems were utilized in the test. Both system types and 
physical systems were utilized based on stakeholder request, doctrine, and equipment 
availability. 

The systems architecture was based on doctrine, requests by stakeholders, and 
equipment availability at MCTSSA. Figure 32 is a diagram of a similar C4I SoS 
architecture, used for MC3T, in a 2007 test event. 
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Figure 32. Representative C4I SoS Architecture. 

This illustration depicts the echelons of command seen in a representative C4I SoS test 
event, as well as connectivity between nodes. Green nodes represent notional Marine Air 
Wing and Marine Infantry Division elements at JITC. Yellow nodes represent notional 
Marine Expeditionary Force, Infantry Division, and Artillery Regiment elements at 
MCTSSA. The blue node represents a simulated aircraft at Naval Air Warfare Center 
(NAWC) China Lake. The white cloud represents connectivity through the Defense 
Research and Engineering Network to remote sites. 

2. Baseline System of System Capabilities 

SoS capabilities are capabilities that are achievable by multiple systems working 
together as a whole, but are not achievable by a single system working on its own. The 
FEDOS 2005 test event included fourteen capabilities. Table 8 lists the fourteen SoS 
capabilities tested. 

3. Defining “Old” versus “New” Capabilities 

The Arena models included a function to model the effort required to plan a C4I 
SoS with both “Old” and “New” capabilities. In this context, an “Old” capability is one 
that has been previously evaluated by the test organization. The distinction between 
“Old” and “New” allowed the simulation of the effects of learning over time. The test 
organization familiar with a capability may have been able to reuse previous or modify 
previous work products. For a “New” capability, the test organization had no reusable 

work products, and may have required some training to exercise the capability correctly. 
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As a result, new capabilities historically take longer to prepare for, regardless of the 


systems involved. 


SoS Capability 

Description 

Blue Position 

Location Information 
(Blue PLI) 

This capability exercises the SoS ability to receive and display all 
updated unit positions for maintaining the Blue Force Common 
Operational Picture (COP) 

Close Air Support 
(CAS) 

This capability exercises the SoS ability to introduce a Joint Tactical Air 
Request (JTAR) into the SoS, approve a mission, and route that mission 
between AFATDS and TBMCS 

Call for Fire (CFF) 

This capability exercises the SoS ability to transmit a Fire Request from 
a Portable Forward Entry Device (PFED) to an Artillery battery via 
AFATDS. 

Red Track Processing 

This capability exercises the SoS ability to use MAGTF intelligence 
systems to process and fuse enemy location (Red Track) information 

Time Sensitive 
Targeting (TST) 

**NEW** This capability exercises the SoS ability to use Fires and 
Intelligence systems to process a Mission Report. 

Air Tasking Order 
(ATO) 

This capability exercises the SoS ability to generated and disseminated 
an ATO throughout the AFATDS network to each Effects Management 
Tool (IOW VI) using the MEB TBMCS Lite 

Battlefield Geometry 
Exchange (BGE) 

This capability exercises the SoS ability to exchange Battlefield 
Geometries and Overlays between the IOS VI and AFATDS 

Common Logistics 

**NEW** This capability exercises the SoS ability to pass logistic 
requests between MEB and RLT CLC2S Server in untethered mode 

EPLRS Time Slot 
(ETS) 

This capability exercises the SoS ability to compare the network 
utilization of the EPLRS 2ms and 4ms time slots. Based on network 
load 

COP Synch Tools 
(CST) 

This capability exercises the SoS ability to disseminate tracks within the 
Global Command and Control System (GCCS) architecture, validating 
that the GCCS - Joint (GCCS-J), IOS VI, and GCCS-COP (Unit 
Operations Center) synchronized their databases. 

Filters and 

Permissions 

**NEW** This capability exercises the SoS ability to filtered track 
information utilizing Geo-filters. 

Message Exchange 
(ME) 

This capability exercises the SoS ability to successfully passed Internet 
Relay Chat and Veriable Message Format (VMF) messages between 
company, battalion, and regiment. 

Overlay Exchange 

This capability exercises the SoS ability pass VMF K05.17 overlay files 
message within the Command and Control Personal Computer (C2PC) 
network. 

Theatre Ballistic 
Missile (TBM) 

**NEW** This capability exercises the SoS ability to generate a missile 
track for dissemination throughout the CST and C2PC gateways. 


Table 8. SoS Capabilities Tested during FEDOS ’05. 

This table lists the 10 old and 4 new SoS capabilities used, at various command echelons, 
in the FEDOS 2005 test event. The capabilities used in the test were defined based on 
stakeholder requirements, doctrine, and equipment availability. 
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E. SIMULATION RESULTS 

1. Arena Simulation Results 

Elasticity - “A measure of responsiveness. The responsiveness of behavior 
measured by variable Z to a change in environment variable Y is the 
change in Z observed in response to a change in Y.” [About.com, 2007], 

Arena was used to calculate elasticity for each of the alternatives. In order to 
accomplish this, the team captured the percent change in output (Variable Z) divided by 
percent change in input (Variable Y). To vary the input, the team increased or decreased 
the input variables (Number of Systems, Number of New Capabilities, and Number of 
Old Capabilities) in 25% increments and recorded the resulting output variables (duration 
and total labor hours). From the baseline of 10 old and 4 new capabilities, and 19 
systems, the alternatives were scaled down in 25% increments to 50% of the original 
scope, and up in 25% increments to 150% of the original scope. The formulas for the 
calculation of the elasticity of duration and labor are given by 


Elasticity of Duration = 

Z (Defined as % Change in Duration) 
Y (Defined as % Change in Input) 

(1) 

Elasticity of Labor = 

Z (Defined as % Change in Labor) 

Y (Defined as % Change in Input) 

(2) 

After an alternative performance against the baseline C4I SoS was completed, the 

baseline performance was recorded. The C4I SoS was scaled down and up, and elasticity 


figures recorded at each 25% change in scope. Elasticity of Duration and Elasticity of 
Labor were calculated using the formula above, and the results for each change in SoS 
were recorded. Mean elasticity was calculated by adding the four elasticity figures and 
dividing the sum by four. 

Table 9 shows the results of incremental and mean calculations for each 
alternative. 
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Average 


Average 


Elasticity 

Elasticity 

Elasticity 

Elasticity 


of 

of 

of 

of 

Alternative 

Duration 

Duration 

Labor 

Labor 

Baseline (FEDOS) 

Baseline reduced by 50% 

0.867 


0.867 


Baseline reduced by 25% 

0.867 

0.867 

0.867 

0.867 

Baseline increased by 25% 

0.867 

0.867 

Baseline increased by 50% 

0.867 


0.867 


MC3T 

MC3T reduced by 50% 

0.782 


0.782 


MC3T reduced by 25% 

0.782 

0.782 

0.782 

0.782 

MC3T increased by 25% 

0.782 

0.782 

MC3T increased by 50% 

0.782 


0.782 


FCB 

FCB reduced by 50% 

0.717 


0.717 


FCB reduced by 25% 

0.717 

0.717 

0.717 

0.717 

FCB increased by 25% 

0.717 

0.717 

FCB increased by 50% 

0.717 


0.717 


SCR 

SCR reduced by 50% 

0.979 


0.979 


SCR reduced by 25% 

0.979 

0.979 

0.979 

0.979 

SCR increased by 25% 

0.979 

0.979 

SCR increased by 50% 

0.979 


0.979 


JTEM - CTM 

JTEM reduced by 50% 

0.831 


0.831 


JTEM reduced by 25% 

0.831 

0.831 

1.661 

1.038 

JTEM increased by 25% 

0.831 


0.831 


JTEM increased by 50% 

0.831 


0.831 



Table 9. Elasticity Results for All Alternatives. 


Elasticity calculations made from the output of the Arena Process Analyzer. Calculations 
for the individual elasticity associated with a change in the C4I SoS are displayed in the 
“Elasticity of Duration” column. Mean Elasticity is calculated by summing the four 
figures for the alternative and dividing by four. Within the limits of rounding, all 
alternatives displayed a consistent elasticity across the scale changes of the SoS. The 
only exception was the JTEM CTM alternative, which showed a significant elasticity 
when decreased to 75% of the original SoS, and otherwise exhibited the same elasticity 
(0.831) at all other ranges. 
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2. Power Simulation Results 

POW-ER was used to generate the date for the “Days to Plan” EM. After 
simulating the performance of each alternative, the results of the simulations were 
calculated and expressed as a number of 8-hour days, using the organization and 
resources assigned to each alternative. Table 10 illustrates the calculated duration of the 
planning process for each alternative. 


Alternative 

POW-ER Estimation of Days to Plan a C4I Evaluation 

Baseline (FEDOS) 

140 Days 

MC3T 

121 Days 

JTEM - CTM 

127 Days 

FCB 

73 Days 

SCR 

158 Days 


Table 10. POW-ER Simulation Results for JC3M Alternatives 

This table provides the results of POW-ER Simulation and displayed the estimated total 
number of days each alternative requires to plan the C4I SoS evaluation. This data is a 
critical EM, and will serve as input to the AoA. 

3. Results of Model Validation 

The Labor Hours results from both the POW-ER and Arena models were 
validated against the FEDOS 2005 results. Table 11 below shows the Labor Hours 
reported from POW-ER and Arena as well as from the FEDOS empirical data. As shown 
in Table 11, the table shows that the Labor Hours reported by Arena was within 1.0 % 
when compared to the historical source data, and the hours reported by POW-ER were 
exactly the same as the FEDOS Labor Hours. The duration of the planning phase of the 
evaluation, as reported by Arena, was at 102% of the actual duration of the historical 
event, within the ±4% range set by the team. The duration as reported by POW-ER was 
96.55% of the actual, and within the ±4% range set by the team. This provided the team 
with a high level of confidence that the models accurately represented the FEDOS 2005 
processes, and that the models were realistic representations of those processes. 
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LABOR HOI 

URS 

RESULTS 

Alternative 

Known Input 

Known Output 
(Labor Hours) 

Arena Output 
(Labor Hours) 

POW-ER Output 
(Labor Hours) 

Baseline 

(FEDOS) 

19 Systems, 

4 New Capabilities, 
10 Old Capabilities 

6,482 

6,484 

(Validated) 

6,482 (Validated) 

DURATION 

RESULTS 

Alternative 

Known Input 

Known Output 
(Duration) 

Arena Output 
(Duration) 

POW-ER Output 
(Duration) 

Baseline 

(FEDOS) 

19 Systems, 

4 New Capabilities, 
10 Old Capabilities 

145 Days 

148 Days 
(Validated) 

140 Days 
(Validated) 


Table 11. Validation of Labor Hours. 

POW-ER and Arena models were validated by ensuring the simulation results were 
within one percent of the known data from historical sources. 


F. SUMMARY 

M&S enabled the team to populate the JC3M Scoring Matrix with data from the 
simulation results. The matrix is the raw data that is needed for the AoA, which enables 
decision-makers to select the best alternative from a utility curve. Table 12 illustrates the 
results provided by M&S. 



Percentage 
of Traceable 
Measures 

Days to 
Plan 

Evaluation 

Quality of 
Planning 
Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

Alternatives 

1.3.2 

1.4.3 

1.4.3 

4.1 

4.1 

FEDOS 


140 days 


0.87 

0.86 

MC3T 


121 days 


0.78 

0.78 

ITEM CTM 


73 days 


1.04 

0.83 

FCB 


158 days 


0.97 

0.97 

SCR 


127 days 


0.71 

0.71 


Table 12. M&S Contribution to JC3M Scoring Matrix. 

Evaluation Measures used for comparing alternatives generated by M&S include 
elasticity and duration. Labor hours (not shown) are used to calculate LCCE. 
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V. LIFE CYCLE COSTS 


A. JC3M LIFE CYCLE COST OVERVIEW 

1. Introduction 

LCCE was a critical component of the System Engineering Design Process, 
because it allowed decision makers to quantify the costs of each alternative solution and 
compare them to other system engineering parameters such as: percentage of traceable 
measures, days to plan evaluation, quality of planning outputs, and elasticity of labor and 
duration. Decision makers could then assess the risks and benefits of implementing each 
of the JC3M alternatives. This chapter will explain the cost estimation methods 
considered, the estimation approach used, how cost data was collected, and provide an 
analysis of the results. 

The LCCE provided an estimate of the total life cycle cost for each of the five 
alternative JC3M solutions over a ten year period. JC3M could be implemented and used 
for more than ten years, or be replaced sooner. The team expected a new C4I SoS 
evaluation system to be developed in order to provide additional evaluation capability in 
response to changing conditions, technology, and doctrine, and chose ten years as a 
representative duration for JC3M. 

The LCCE was limited to estimating the costs of planning a C4I SoS evaluation. 
The design of JC3M assumed the use of existing processes for both the conduct and 
reporting of C4I SoS evaluations; test organizations implementing the JC3M system were 
not expected to see cost changes for these processes. The cost of buildings, C4I hardware 
and software, and supporting infrastructure required to conduct C4I SoS evaluations were 
not considered in the LCCE. These items were considered to be essential resources for 
any organization already conducting C4I SoS evaluations, and were considered sunk 
costs, i.e. a cost incurred in the past that will not be affected by current or future decisions 
[OMB Circular A-94, 1992], Sunk costs were not considered in the LCCE or comparison 
of alternatives [Thuesen and Fabrycky, 2001: 24], 
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The life cycle phases of the JC3M system, used in the LCCE, were Development 
and Implementation, Operations and Support, and Transition and Disposal. 

The development phase included interaction with stakeholders to begin the 
requirements process, designing the functional architecture, modeling, trade studies, and 
analysis of alternatives [Buede, 2000]. Implementation was the phase in which the 
alternative solution was tailored for use under local operating conditions. Operations and 
Support (O&S) was the phase in which operations, maintenance, and support of the 
JC3M systems and associated support equipment took place. Transition and Disposal 
was the phase in which JC3M would be retired. 

In generating the LCCE, a mix of personnel, infrastructure, and facilities was 
stipulated. The resource pool identified to conduct JC3M alternatives consisted of seven 
Senior System Analysts who identified system capabilities and defined and documented 
test cases; six Senior Tech Specialists who planned, designed, coordinated, and controlled 
the evaluation; and one Senior Program Manager who managed and coordinated the 
project and program activity. Physical facilities included laboratory spaces, network 
connectivity and infrastructure, and adequate communications. An organization with a 
significantly different resource pool or physical plant will experience different O&S costs 
than what was estimated in this project. 

a. Development and Implementation Phase 

The Development and Implementation phase included the design, 
development, and procurement of systems and support items necessary to conduct 
planning of C4I SoS evaluations. In estimating the cost of implementation for the 
alternatives, published cost estimation information and interviews with SME were used to 
obtain the necessary information. Interviews were performed with MCTSSA Resource 
Manager [Manning (c), 2007], Architecture Branch Head [Nguyen, 2007], Data Analysis 
Lab Manger [Chance, 2007], FEDOS SME [Hoesly, 2007], MC3T IPT Lead [McLean 
(b), 2007] and Technical Lead [Finn, 2007], These interviews along with Tec hn ical 
Support Plans [Manning (b), 2005] and Consolidated Requirement Assessment [McLean 
(a), 2007] were the source of the baseline and MC3T cost estimates. Based on SME 
interviews, historical financial data from the 2005 FEDOS test event, and monitoring the 
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progress of MC3T at MCTSSA, the team determined that the development and 
implementation phase of the JC3M system would last approximately one year. 

b. Operation and Support Phase 

The Operation and Support (O&S) phase included operation, maintenance, 
and support of the systems and associated support equipment over the life of the system. 
As noted, the LCCE only included the costs of planning a C4I SoS evaluation. The O&S 
phase was determined to have a duration of nine years from the year of implementation. 
The team expected that by the end of the ninth year the JC3M system would be replaced 
as legacy systems are retired and new systems are designed and developed as part of the 
SoS. As the JCIDS process continues to develop, more systems should be “bom joint” 
and their capabilities, as part of the C4I SoS, should be understood. JC3M will need to 
be continually evaluated and updated to accommodate changes in technology, system and 
SoS operations, and concepts of employment. 

c. Transition and Retirement Phase 

The Transition and Retirement phase included costs associated with the 
termination and retirement of the process. If the O&S phase was extended beyond, or 
reduced from, nine years this phase would take place later or earlier, but was still 
projected to last approximately one year. The team estimated that the cost of this phase 
would be minimal due to the conceptual nature of the activities conducted by the JC3M 
system. The three phases of the LCCE are graphically displayed in Figure 33. 
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Figure 33. Phases of JC3M Life Cycle. 

The three life cycle phases of the JC3M alternatives under consideration are 
Development and Implementation, Operations and Support, and Transition and 
Retirement. This graphically displays the phases over the ten year life cycle. 
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2. 


Life Cycle Cost Elements 

a. Development and Implementation Costs 
The development and implementation cost for JC3M was treated as a 

non-recurring cost. The targeted customers of the JC3M system were assumed to be DoD 
organizations involved with C4I SoS test and evaluation. The cost for developing and 
implementing JC3M came primarily from process documentation and initial training. 
Because of the conceptual nature of JC3M, there were no costs expected from the 
procurement of software packages, software licenses, or hardware additions or upgrades. 

b. Operation and Support Costs 

O&S cost is the major portion of the life cycle cost of JC3M. The O&S 
Cost-Estimating Guide [Cost Analysis Improvement Group, 2002] defines O&S cost as 
the “costs of personnel,... services, and sustaining support....” associated with any 
system. To arrive at a realistic estimate of JC3M over its life cycle, current information 
was gathered from a variety of interviews (see Section A.l.a “Development and 
Implementation”) and financial data from the FEDOS TSP [Manning (b), 2005] and 
MC3T CRA [McLean (a), 2007], internal MCTSSA documents used to forecast and 
manage the respective projects. The O&S cost elements included the labor cost of 
personnel required to plan a C4I SoS evaluation; follow on training; and the software and 
hardware periodic upgrades, license renewals, and maintenance contracts required for 
planning C4I SoS evaluation. 

c. Transition and Retirement Costs 

The transition and disposal cost of the JC3M system consisted of the 
manpower costs of personnel required to provide customer support during the transition 
period, document archiving, and configuration management. Customer support was 
estimated to require the services of at least one engineer to work with users to facilitate 
the teardown and disposal of systems and at least one configuration management 
specialist to assist in archiving relevant documents and software. This phase was 
estimated to last no more than one year after the end of the O&S phase of the JC3M life 
cycle. The majority of JC3M documents would be electronically stored and managed; 
therefore, costs in this phase are associated with computers and online storage space. The 
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cost to transition and retire the planning processes associated with JC3M are considered 
to be minimal due to the conceptual nature of JC3M. 

3. Cost Estimation Assumptions and Constraints 

The cost estimate assumptions and constraints are summarized as follows: 

■ The application of JC3M processes did not require any additional 
equipment or tools. 

■ JC3M planning processes are conceptual and no specialized software or 
hardware was required. 

■ Specialized education or training was not required in applying the JC3M 
process. 

■ The cost for developing and implementing JC3M was a one-time non¬ 
recurring cost. 

■ The cost for the disposal phase was minimal due to the conceptual nature 
of JC3M. 

■ Overhead cost in each organization was not considered. 

■ The team recognized the variance in duration of “Days to Plan” among the 
alternatives, but for consistency, elected to consider each alternative 
conducting one C4I SoS evaluation per year. 

B. COST ESTIMATION METHODS CONSIDERED 

The JC3M LCCE was completed using a combination of the Analogy cost method 
[Defense Acquisition Guidebook, 2006] and the Activities-Based Costing (ABC) method 
[Blanchard and Fabrycky, 1998: 580]. These methods are described below. 

Estimation by Analogy is employed when the project under consideration is 
similar to a previously fielded or implemented system. Estimation by analogy requires 
that current systems similar in design and function are selected for the basis of cost 
estimation [Defense Acquisition Guidebook, 2006], 

MC3T is a system designed for defining, documenting, and evaluating the 
performance of a MAGTF C4I SoS as a single system, in accordance with modem 
systems engineering practices. Because JC3M was designed to perform functions similar 
to those of MC3T, the team determined that analogy was an appropriate method of cost 
estimation. Historical cost data from the MC3T system was used to estimate the cost of 
development and implementation of MC3T. MC3T cost data was obtained primarily 
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from interviews with the MC3T IPT Lead McLean (a) [2007], and the draft CRA 
[McLean (b), 2007], 

ABC is used when estimating the cost to carry out the activities and processes of a 
system. The main goal of this methodology is to ensure that all costs are traceable back 
to the functions or processes that actually occurred within the system. In the ABC 
methodology, costs are directly traceable to the applicable cost-generating process and 
product. Cost can be easily estimated on a functional basis, and the emphasis is on 
resource consumption. Processes and products are performed by activities, and activities 
consume resources. The ABC approach fosters the establishment of cause-and-effect 
relationships, enables the identification of the high-cost contributors, and eliminates 
double counting that is typically inherent with the application of direct and indirect costs. 
[Blanchard and Fabrycky, 1998: 580-581], The ABC method was used to estimate O&S 
cost for the JC3M alternatives. 

The JC3M system utilized resources to execute functions and within these 
functions processes and tasks were executed in order to produce the desired outputs. The 
cost of each process or task, in labor-hours consumed, was directly traceable back to the 
cost-generating function. The input to the cost model was based on the labor hours 
required to execute the functions associated with each of the alternatives. 

The team also considered the Engineering Estimate method and the Actual Cost 
method [Defense Acquisition Guidebook, 2006], The engineering estimation method is 
employed when the system is broken down into components. However, since the JC3M 
system consists primarily of activities, this methodology was not suitable. The Actual 
Cost Estimation method was considered but because there was no actual cost experience 
or trends from prototypes, engineering development models, or early production items to 
project estimates of future costs this method was also rejected. 
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C. JC3M COST ESTIMATION APPROACH 

The team followed the cost estimation approach described by Blanchard and 
Fabrycky [1998: 567] to estimate the costs for the JC3M system. The approach was 
tailored to fit the requirements of the JC3M system as defined in Chapter II “Problem 
Refinement.” 

The cost model was designed to estimate the life cycle cost of planning C4I SoS 
evaluations as conducted by each of the alternative solutions. The cost model was 
constructed using Microsoft Excel® spreadsheet software. The cost model calculated the 
cost of each task, year, life cycle phase, and overall cost for each alternative, and the 
costs for each alternative were summarized. 

In each phase of the JC3M life cycle, tasks and sub-tasks were executed in order 
to support the functions of the system. These tasks and sub-tasks were identified based on 
the functional decomposition of the system. The output values from the LCCE provided 
the cost inputs to the AoA. 

The JC3M Cost Breakdown Structure (CBS) used a hierarchical framework for 
estimating the life cycle cost. The CBS provided a structure that identified the elements 
or categories of costs that were incurred in each phase of the JC3M life cycle. It also 
served as a guide for cost data collection for each of the high level functions of the JC3M 
system. Because each of the alternatives was unique, not all the alternatives had cost data 
for the identified cost categories. The FEDOS alternative did not define performance 
measures, and did not report a cost for this task. 

The CBS defined the specific elements included in the cost estimate and 
organized the LCCE results in a hierarchy [Defense Acquisition Guidebook, 2006], The 
CBS identified the cost elements for each phase and guided cost data collection for each 
of the high level functions of JC3M. Figure 34 is a graphical representation of the JC3M 
CBS. 
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Figure 34. Cost Breakdown Structure of JC3M. 

The CBS provided a structure that identified the categories of costs in each phase of the 
JC3M life cycle. It also served as a guide line for cost data collection for each of the high 
level functions of the JC3M system. 

D. COST DATA COLLECTION 

Data for this effort was extracted from the MCTSSA CRA [McLean (a), 2007] 
that provided resource requirement assessments for funding, staffing, data collection and 
monitoring networks, data capture, and information assurance. The O&S cost of JC3M 
was most influenced by the direct labor costs of the technical staff. 


The MCTSSA test team consisted of federal civilian employees, contractors, and 
a small military staff that provided C4I systems support, operation, and management. 
Fully burdened labor rates for civilians and contractors were provided by the MCTSSA 

comptroller office for Fiscal Year (FY) 2007 [George (a), 2007]. Fully burdened FY 

88 















































2007 labor rates [Military Composite Pay and Reimbursements Rate Table, 2007] were 
used to calculate cost of military personnel. 

The MC3T IPT lead [McLean(b), 2007] was interviewed to obtain information on 
the duration of the development, implementation, and operations of MC3T. Projected 
personnel labor hours, software and hardware cost, as well as the number of billets 
required were obtained from the draft version of the CRA provided by the MC3T IPT 
Lead [McLean (a), 2007], The CRA was the primary source of data for the cost 
estimation. The team also interviewed stakeholders and subject matters experts to obtain 
cost data and information that was not readily available. In calculating the personnel 
labor costs, the fully burdened rate for government civilians [DoD Civilian Personnel 
Management Service, 2007], military [Office of the Under Secretary of Defense for 
Personnel and Readiness, 2007], and contractor billets [GSA schedule for Systems 
Engineering Support Service] were used. Interviews with the MCTSSA assistant 
comptroller [George (b), 2007] were also conducted to provide additional information on 
the fully burdened labor rates for civilians and contractors at MCTSSA. 

E. ANALYSIS OF COST DATA 

1. Development Costs 

Development costs for the FEDOS alternative were recorded as $0 in the LCCE, 
because this alternative was fully developed. Development costs for the MC3T 
alternative were recorded as $0 in the LCCE, because this alternative was considered 
fully developed. Resources used for development of the MC3T alternative were 
identified, based on review of historical data and SME interviews, and used by analogy to 
estimate the development costs of both the FCB and SCR alternatives which did not have 
historical development costs. FCB and SCR estimated developments costs are adjusted 
to account for specific differences between the alternatives. Development costs for the 
ITEM CTM alternative were based on SME interviews, and are described below. All 
development costs are recorded in the LCCE and summarized in Table 13. 

Development of MC3T included a Self Assessment Team (SAT), a 
multidisciplinary IPT formed to evaluate MC3T processes and assist in refining processes 
prior to execution. The SAT also served to capture lessons learned and document 
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changes for process improvement. The estimation of the labor cost for the SAT was 
based on the MC3T POAM-OOl-Vl [McLean (c), 2007], the Requirements for Members 
of the MCTSSA Self Assessment Team, [Villar (a), 2007] and an interview with the SAT 
lead [Villar (b), 2007], 

Colonel Eileen Bjorkman, United States Air Force (USAF), Test Director, Joint 
Test Evaluation Methodology, stated the JTEM CTM will not be fully developed until 
FY09, and estimated the total remaining cost to develop the CTM through 2009 at $3.5M 
[Bjorkman, (b) 2007], This estimate includes JTEM CTM Community of Interest work, 
documentation and process writing, and participation in events. The JTEM CTM 
development cost does not consider overhead cost such as utilities, facilities operation 
and maintenance, administration costs, and other overhead cost not directly related to 
labor cost. This estimation is consistent with the cost estimations used for development of 
the other alternatives. This development cost is allocated in the cost model as $1.03M for 
the remainder of year 1 (FY07), and $2.47M for year 2 (FY08). 

2. Implementation Costs 

Implementation costs for the FEDOS alternative were not found in historical data. 
Implementation costs for the FCB, SCR, and JTEM CTM alternatives were not available 
because they have not been used. The cost to implement these four alternatives was 
estimated by analogy with the MC3T implementation costs, because historical data is 
available, the alternatives perform similar tasks, and are of similar complexity. All 
implementation costs are recorded in the LCCE and summarized in Table 13. 

The CRA [McLean (a), 2007] provided the most accurate source of data on 
personnel, data capture, hardware, and software costs of MC3T implementation. This 
estimate includes costs of training personnel, documentation, capturing lessons learned, 
documenting changes, and software licensing fees, and totals approximately $1.1M. The 
implementation effort will take MC3T to the start of the first evaluation; the conduct of 
the initial C4I SoS evaluation is not included. 

Implementation of the JTEM CTM does not begin until year 3 of the LCCE, when 
development of the alternative is completed. 
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3. Results of Cost Data Analysis 

The JC3M team determined the most expensive alternative was the FCB 
alternative, at a cost of $8.13M over the ten-year projected lifecycle. The team also noted 
that FCB is approximately $0.41M more than SCR. The team calculated the cost of FCB 
as a cost to DoD, i.e. while the senior SMEs who generate the performance measures do 
not charge their efforts directly to a C4I test organizations, their time and effort is a cost 
to DoD. In order of increasing cost, the other alternatives were FEDOS at $5.01M; 

MC3T at $5.97M; JTEM CTM at S6.97M; and SCR at S7.72M. 

The team determined that MC3T was estimated to cost approximately $0.96M 
more than FEDOS, which it has replaced. While this is a significant cost difference, the 
increase can be directly attributed to the increased in scope, duration, and level of effort 
involved in MC3T, which, in addition to the factual data cited above, anecdotally 
supported the increased cost of MC3T. 

The team noted the alternative with the lowest estimated O&S cost was the JTEM 
CTM alternative, at $2.3M, followed by FEDOS at $3.9M, MC3T at $4.8M, SCR at 
$5.6M, and FCB at $5.8M. This is significant because O&S represents the largest 
portion of the LCCE, and is the portion directly assumed by a test organization that 
implements any alternative. 

Implementation of JC3M would modify topics assessed in a C4I SoS evaluation, 
which could cause cost changes in the conduct and reporting phases of the evaluation. In 
part to assess these potential changes, the team recommended validation of JC3M results 
(see Chapter 7, Section D “Conclusion”) through a “real world” evaluation. 

The team noted that due to the continuing development of the JTEM CTM 
alternative, the O&S phase would not start until one year after the other four alternatives. 
As a result, over the ten year duration of the LCCE, the JTEM CTM would complete one 
less C4I SoS evaluation. Table 13 summarizes the LCC results. 
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Life-Cycle Year 

Total Cost 

Alternatives 

1 

2 

3 

4...9 

10 

($) 

FEDOS 







Development 

0 

0 

0 

0 

0 

0 

Implementation 

1,052,527 

0 

0 

0 

0 

1,052,527 

Operational & Support. 

0 

419,497 

419,497 

419,497 

2,200 

3,908,178 

Transition and Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total Cost 

1,052,527 

419,497 

419,497 

419,497 

52,200 

5,010,706 

MC3T 







Development 

0 

0 

0 

0 

0 

0 

Implementation 

1,169,414 

0 

0 

0 

0 

1,169,414 

Operational & Support. 

0 

525,537 

525,537 

525,537 

2,200 

4,756,500 

Transition and Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total Cost 

1,169,414 

525,537 

525,537 

525,537 

52,200 

5,975,913 

JTEM CTM 







Development 

1,030,000 

2,470,000 

0 

0 

0 

3,500,000 

Implementation 

0 

0 

1,169,414 

0 

0 

1,169,414 

Operational & Support. 

0 

0 

0 

558,535 


2,253,410 

Transition and Disposal 

0 

0 

0 

0 


50,000 

Total Cost 

1,030,000 

2,470,000 

1,169,414 

558,535 

52,200 

6,972,824 

FCB 







Development 

1,021,835 

0 

0 

0 

0 

1,021,835 

Implementation 

1,301,282 

0 

0 

0 

0 

1,301,282 

Operational & Support 

0 

650,223 

650,223 

650,223 

2,200 

5,753,985 

Transition and Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total Cost 

2,323,117 

650,223 

650,223 

650,223 

52,200 

8,127,101 

SCR 







Development 

952,007 

0 

0 

0 

0 

952,007 

Implementation 

1,169,414 

0 

0 

0 

0 

1,169,414 

Operational & Support. 

0 

624,451 

624,451 

624,451 

2,200 

5,547,811 

Transition and Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total Cost 

2,121,421 

624,451 

624,451 

624,451 

52,200 

7,719,232 


Table 13. Life Cycle Cost Summary. 


Summary sheet of life cycle costs for all alternatives based on CBS over a 10-year life 
cycle 
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VI. ANALYSIS OF ALTERNATIVES 


A. METHODOLOGY 

The team conducted an Analysis of Alternatives in order to compare the 
performance of each of the five alternatives described in Chapter III, Design Alternatives. 
AoA compared the outputs of EMs and LCCE for each alternative. The team utilized an 
approach that provided a comparison of total cost to total performance. 

An Analysis of Alternatives (AoA) compares alternatives by estimating 
their ability to satisfy the identified requirements through an effectiveness 
analysis and by estimating their life cycle costs (LCC) through cost 
analysis. The results of these two analyses are used together to produce a 
cost-effectiveness comparison that allows decision makers to [evaluate] 
cost and effectiveness simultaneously [Feuchter, 2004: 7], 

B. MULTI-ATTRIBUTE UTILITY THEORY 

Multi-attribute Utility Theory (MAUT) is a “quantitative method for aggregating 
a stakeholder’s preferences over conflicting objectives to find the alternative with the 
highest value when all objectives are considered” [Buede, 2000: 361], The five 
alternatives under consideration were evaluated and compared against each other in this 
section using MAUT. The goal of the analysis was to provide a structured method in the 
decision-making process. There are several techniques that can be used to compare 
alternatives when there are multiple evaluation measures with different units of measure. 
The team decided to use the Value Modeling technique as part of the JC3M SEDP. 

Sage and Rouse [1999: 1128] stated that to select the preferred alternative it is 
necessary to combine evaluation measures into a single measure of the overall 
desirability of an alternative. This is done by developing a value function 
(v(Xj ,x 2 ,... ,x n ) ). The equation for the value function, under the condition of linear 
additive independence of attributes, is given by 

v(v ,X 2 ,...,X n ) = WjVj (Xj ) + w 2 v 2 (x 2 ) + ••• + w n v n (x n ) ^ 

where xj, xi, ... x„ are the EMs for an objective/requirement, vfxi), v^fxi), ... v n (x„) are 
the utility scores of the EMs for a specific alternative, and wj, W 2 , ... w„ are the weights 
for each EM. 


93 



C. VALUE MODELING TECHNIQUE 

Value modeling as applied to this project was discussed in detail in Chapter II, 
Problem Refinement. Value modeling has three steps, 1) the definition of a Qualitative 
Value Model, 2) the definition of a Quantitative Value Model, and 3) the definition of an 
Additive Value Model. Figure 35 provides an overview of the structure of value 
modeling. 



score of each alternative 


Figure 35. High Level Outline of Value Modeling. 

Value Modeling is composed of Qualitative Value Modeling, Quantitative Value 
Modeling, and Additive Value Modeling. Details of the JC3M Qualitative Value 
Modeling process are provided in the Chapter II “Problem Refinement.” Quantitative 
Value Modeling includes the definition of relative importance of EM; Additive Value 
modeling converts raw scores to total value of each alternative. 

The Critical EMs used to compare performance of each alternative were described 
in Table 1, Chapter II, Problem Refinement. The EMs used in the functional analysis 
were the same EMs used to evaluate the alternatives. This provided clear traceability 
between the functional requirements and the AoA. In order to perform an effective 
evaluation of performance it was necessary to have a consistent and accurate set of 
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criteria. These criteria must be based on the design objectives, i.e., what it is that the 
design is meant to achieve [Cross, 2000: 140]. The criteria used to evaluate EMs for the 
alternatives are shown in Table 14. Table 14 expands on Table 1 by adding the ideal 
values of the EMs. 



Percentage of 
Traceable 
Measures 

Days to Plan 
Evaluation 

Quality of 
Planning 
Outputs 

Elasticity of 
Labor 

Elasticity of 
Duration 

JC3M 

Function 

Define 

Measures 

1.3.2 

Planning 

Results 

1.4.3 

Planning 

Results 

1.4.3 

Input System 
Flexibility 

4.1 

Input System 
Flexibility 

4.1 

Definition 

Alternative 
generated 
measures, 
traceable to 
stakeholder 
requirements, 
divided by the 
number of 

measures 
generated by 
the alternative. 

Sum of labor 
hours required, 
expressed in 
days, to plan 

C4I SoS 
evaluation 

Assign an 
overall quality 
level to the 
planning 
documents 
produced. 

Divide percent 
change in labor 
hours to 
conduct 
planning phase 
of JC3Mby the 
percent change 
in systems 
under test. 

Divide percent 
change in 
duration to 
conduct 
planning phase 
of JC3M by the 
percent change 
in systems 
under test. 

Range 

0-100% 

>0 

Likert Scale of 

1 -4 

0 - oo 

0 - oo 

Ideal Value 

100% 

Less is better 

4 is best 

Less is better 

Less is better 


Table 14. JC3M Critical Evaluation Measures. 

Critical EMs used to compare the performance of alternative solutions. This table 
expands on Table 1 by including ideal values. 

D. QUANTITATIVE VALUE MODEL 

The Quantitative Value Model utilized the decision maker’s preferences with 
respect to EMs. This involved defining utility curves and weights for each EM. Utility 
curves were used to normalize EM scores, i.e., convert a raw score received from M&S 
or from an SME, into a utility score. For example, consider the top speed of a vehicle as 
an EM of interest to a decision maker. If a vehicle being considered for purchase had a 
top speed of 60 mph, this raw value would be converted to a value on a scale of 0 to 1, 
where 0 represents no utility to the stakeholder, and 1 represents the highest utility to the 
stakeholder. The “worth” of the top speed of 60 mph would be dependent on the 
decision maker’s preferences and could be modeled with a utility function. The utility 

score of EMs were expressed on a common scale in order to allow useful comparisons. 
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There may be multiple EMs and each may have different relative worth to a 
decision maker. In the example above, purchase price may be more important to a 
decision maker than top speed, braking distance, or acceleration. Based on the 
preferences of the decision maker(s), appropriate weights were assigned to each EM in 
order to reflect their relative importance. 

1. Eliciting Utility Curves from Stakeholders 

The team evaluated two methods to implement utility functions, also referred to 
as Standard Scoring Functions (SSFs). [Wymore, 1993] developed a set of twelve SSFs. 
In addition, [Buede, 2000:365] described a family of exponential utility functions. 
[Daniels, 2001:209] compared both methods and concluded that only eight utility 
functions are required for all applications that he and his team had encountered. Four of 
the eight are Wymorian SSFs. The team selected two Wymorian functions: SSF 3 and 
SSF 9, because they fit the parameters that can be assignable to the evaluation measures 
being modeled by the team [Daniels, 2001:206], 

The SSF3 guideline is: “If more is better, the customer can provide both a finite 
upper bound and a finite lower bound, then choose SSF3. This is by far the most common 
scoring function used when dealing with an EM where more is better” [Daniels, 2001: 
204], 

The SSF9 guideline is: “If more is worse, and the stakeholder can provide both a 
finite upper bound and a finite lower bound, then choose SSF9. This scoring function is 
by far the most frequently used with EMs where more is worse” [Daniels, 2001: 207]. 

The five parameters required to create Wymorian SSFs are defined as follows: 

L - The lower threshold of performance for the EM below which the value to the 
customer is undesirable (but not necessarily unacceptable) and is assigned a zero score. 

B - The parameter B is called the baseline value for the EM and can be chosen as 
the design goal or the status quo for this or similar systems. By definition, baseline values 
are always assigned a score of 0.5. 

U - The upper threshold of performance for the EM above which the value to the 
customer is assigned a score of one. 
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S - The parameter S determines the behavior of the scoring function in the 
neighborhood of the baseline value B. Mathematically, S is the slope of the tangent to the 
scoring function at the baseline value B. The slope represents the maximum incremental 
change in the customer’s quantitative judgment with each incremental change in input. 

D - The parameter D represents the domain of definition of the scoring function. 
D is defined as the range between L and U. Conceptually, D clearly states the range of 
input values that are possible from a build-ability viewpoint or legal due to mandatory 
requirements. Values outside this range constitute impossible or unacceptable inputs 
[Daniels, 2001: 203]. 


Each SSF figure included the SSF Value or Utility Score along the y axis and the 
raw EM score along the x axis. In addition, each figure included the parameters that can 
be used to recreate the SSF. Figure 36 illustrates Wymore’s SSF 3 and SSF 9. 



Figure 36. Wymorian Standard Scoring Functions 3 and 9. 

SSF 3 and SSF 9 were chosen by the team to use during AoA. Five parameters were 
needed to create a Wymorian SSF. These are Lower Limit (L), Baseline Value (B), 

Upper Limit (U), Slope (S), and Domain (D). SSF 3 was used to score measures that are 
more valuable as they increase (Percentage of Traceable Measures). SSF 9 was used to 
score measures that are more valuable as they decrease (Days to Plan Evaluation). 

Thomas Fee Rodgers in cooperation with A. Terry Bahill from the University of 

Arizona developed a software program that implements the SSFs described here 

[Rodgers and Bahill, 2007]. The team used this software to implement the SSFs 

described in Figures 38 through 41. 
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The team constructed initial SSFs for the five EMs and presented them to a group 
of SMEs, as described in Appendix C. Representatives from JTEM, JITC, NPS, 
MCTSSA, and NAWC China Lake were invited to provide stakeholder preferences for 
the Quantitative Value Model. Two representatives from MCTSSA, with United States 
Marine Corps (USMC) C4I SoS test experience, and two representatives from NAWC 
China Lake, with United States Navy (USN) C4I SoS test experience, were available and 
consulted to provide stakeholder preferences. Stakeholders approved the SSFs and 
provided weights for each EM using the Analytical Hierarchy Process (AHP). 

2. Percent Traceable Measures Utility Curve 

Percentage of Traceable Measures was an EM that reported on how well the 
alternative creates performance measures traceable to authoritative sources. As the 
percentage increased, the performance measures were more valuable to the organization 
conducting the C4I SoS evaluation. The SSF used to score this EM is illustrated in 
Figure 37. 


Percentage of Traceable Measures 



Figure 37. Utility Function for Percentage of Traceable Measures. 

This is the stakeholder validated Utility Function used for this EM. A higher percentage 
was more desirable. Values less than 65% received a utility of 0. Values greater than or 
equal to 95% received a utility of 1.0. SSF 3 Parameters were L = 65%, B = 80%, U = 
95%, D (U - L) = 30%, and S = 0.67. 
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3. Days to Plan Utility Curve 

The Days to PlanEM reported how many work days were required by the 
alternative to plan the evaluation of the baseline C4I SoS. As the number of days 
decreased, the alternative became more valuable to the organization conducting the C4I 
SoS evaluation, because more evaluations could be planned over the same time period, or 
plans could be performed more quickly. The SSF used to score this EM is illustrated in 
Figure 38. 


Days to Plan 
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Figure 38. Utility Function for Days to Plan Evaluation. 

This is the stakeholder validated Utility Function used for this EM. A lower number of 
days was more desirable. Values less than 90 days received a utility of 1.0. Values 
greater than or equal to 210 days received a utility of 0. SSF 9 Parameters were L = 90 
days, B = 150 days, U = 210 days, D (U - L) = 120, and S = - 0.0167. 

4. Quality of Planning Outputs Utility Curve 

Quality of Planning Outputs was an EM that reported on overall “goodness” of 
the test plans and test procedures produced by the alternative. As quality increased, the 
alternative became more valuable to the organization conducting the C4I SoS evaluation, 
because evaluations become more accurate, more effective, and more efficient. The SSF 
used to score this EM is illustrated in Figure 39. 
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Quality of Planning Outputs 



Figure 39. Utility Function for Quality of Planning Outputs. 

This is the stakeholder validated Utility Function used for this EM. A raw score of 4.0 is 

ideal and received a utility of 1.0. SSF 3 Parameters were L = 0, B = 2.0, U = 4.0, D (U - 

L) = 4.0, and S = 0.5. 

5. Elasticity of Labor and Elasticity of Duration Utility Curves 

Elasticity of Labor was an EM that reported on how the alternative responded to 
changes in the C4I SoS under evaluation. Specifically, the percent change in labor hours 
to conduct planning phase of JC3M was divided by the percent change in input 
parameters (number of SUT, number of new capabilities, number of old capabilities) and 
the resulting ratio was the measure of elasticity. 

Values less than 1.0 are inelastic, and indicated that changes in the C4I SoS under 
evaluation cause a change in the number of labor hours required at a smaller rate. As the 
SoS increased in size, there was less labor required, per system, to evaluate the 
performance of the SoS. Elasticity is valuable when conducting evaluations of C4I SoS 
as large or larger than the candidate test architecture described in Chapter IV section C.2, 
Select a Candidate C4I Test Architecture. The SSF used to score Elasticity of Labor is 
illustrated in Figure 40. 

Elasticity of Duration, also illustrated in Figure 40 is an equivalent EM, and 
measured how the duration (number of days) of the alternative process changed in 
response to changes in the C4I SoS under evaluation. Elasticity of Duration would be 
valuable when conducting evaluations of C4I SoS as large or larger than the candidate 
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test architecture described in Chapter IV section C.2, Select a Candidate C4I Test 
Architecture. 


Elasticity of Labor/Duration 



Figure 40. Utility Function for Elasticity of Labor and Elasticity of 
Duration. 

This is the stakeholder validated Utility Function used for this EM. A lower number for 
Elasticity was desirable. Values less than 0.5 days received a utility of 1.0. Values 
greater than or equal to 1.5 days receive a utility of 1.0. SSF 3 Parameters were L = 0.5, 

B = 1.0, U = 1.5, D (U — L) = 1.0, and S = 2.0. 

Values less than 1.0 are inelastic, and indicated that changes in the C4I SoS under 
evaluation cause a change in the number of days required at a smaller rate. As the SoS 
increased in size, there were less days required, per system, to conduct planning of the 
SoS test. Elasticity is valuable when conducting evaluations of C4I SoS, to determine the 
effects that increases in the number of systems will have on the duration for planning the 
SoS evaluation. Chapter IV section C.2, described the baseline candidate C4I test 
architecture; by adding or reducing the number of systems one can determine the 
elasticity of the number of days required to plan a C4I SoS event. 

E. ANALYTICAL HIERARCHY PROCESS 

Buede [2000: 369] described several methods to elicit weights for calculating the 
relative worth of EM. The team chose the AHP because it is useful for cases where there 
are 3-7 objectives [Saaty, 1994], 

The weights were elicited from three SMEs, using the AHP verbal mode for 
pairwise comparisons. The SMEs provided their assessment of how the EMs compared 
against each other using the verbal scale and the numerical scale in Figure 41. 
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STRONGLY St ronGLY AbSOLUTELY 



Figure 41. Sample Analytical Hierarchy Process Scale. 

This scale ties the numerical AHP values to the verbal equivalents utilized during the 
JC3M stakeholder questionnaire. This sample scale ranges illustrates the range of 
potential values from one ninth as valuable to nine times as valuable. 

A value of 1 signifies the objectives were equal; 3 signifies an objective was 

weakly superior to the other objective; 5 signifies an objective was strongly superior; 7 

signifies very strongly; and 9 signifies absolutely superior. Objective measures are 

compared pairwise, i.e. Percentage of Traceable Measures is strongly superior (5) to 

Days to Plan Evaluation. The AHP then attributed a relative weight of 5 to Percent 

Traceable Measures and a relative weight of 1 to Days to Plan Evaluation. The reciprocal 

comparison is also recorded in the cell on the opposite side of the diagonal; Days to Plan 

Evaluation is 1/5 with respect to Percentage of Traceable Measures. The relative weights 

remain the same; it is only the order of measures that changes. 

This assessment resulted in some inconsistent weighting of the objectives. This is 
not too unusual for the first iteration in trying to quantify the relative importance of the 
different evaluation measures. As this inconsistency was discovered after the SMEs were 
no longer available, the team did not have the opportunity to re-engage the SMEs in order 
to resolve those inconsistencies. The degree of inconsistency of the pairwise comparison 
matrix exceeded 0.1. Nevertheless, the team performed an eigenvector calculation on the 
responses. An eigenvector is a mathematical term encountered when studying linear 
transformations. In AHP, the result of the eigenvector calculation provides the weights 
for each EM. Saaty [1994] demonstrated mathematically that the eigenvector solution 
was the best approach to obtain weights from the AHP matrix. Equation 4 can be treated 
as an eigenvector problem and is solved for w to obtain the weights. 
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Aw = nw (4) 

where A is the AHP matrix, w is the eigenvector and Wj is the weight of EMi, and 
n is the dimension of the matrix and the eigenvector. Equation 4 is the Eigenvalue 
equation expressed as a matrix. 
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( 5 ) 


In Equation 5 solving for w yields the weights for each evaluation measure 
[Ishikawa, 2007], shown in Table 15. 



Percentage 

of 

Traceable 

Measures 

Days to 
Plan 

Evaluation 

Quality of 
Planning 
Outputs 

Elasticity 
of Labor 

Elasticity 

of 

Duration 

EM 

Weights 

0.248 

0.058 

0.419 

0.084 

0.192 


Table 15. AHP Evaluation Measure Weights. 

The weights are calculated using an iterative eigenvector process. 


A sensitivity analysis on the effect of the weights was performed. The conclusion 
is that the noted minor inconsistencies do not invalidate the final results. Based on sound 
reasoning, the results of applying these weights to the utility scores have resulted in valid 
relative rankings of the alternatives even though the absolute values used to populate the 
quantitative decision matrix would differ slightly if the weights were changed. 

F. RAW DATA VALUES 

This section summarizes the raw data values obtained through modeling and 
simulation and offline evaluation. 

1. Results of Percent Traceable Measures EM 

Percentage of Traceable Measures (PTM) was the EM for the Define Measures 
(1.3.2) function of the JC3M value hierarchy. The objective of the Define Measures 
function was to determine how well an alternative identified measures of performance 
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(MOP) when evaluating the SoS. An EM should not be confused with a MOP. EMs are 
measures created by the team to gauge functions of the JC3M system. MOPs are 
measures that gauge performance of a C4I SoS. The detailed methodology used to derive 
the PTM EM is included in Appendix G. The results are also summarized in this section. 

Definition of PTM: 


This EM was calculated by dividing the number of measures (traceable to 

stakeholder requirements) generated by the alternative, by the number of measures 

generated by the alternative 

, # Traceable Measures Created 

PTM =- 

# Total Measures Created 


( 6 ) 


However, the team came to the conclusion that it was not feasible to calculate the 
PTM EM as it was defined because that would entail exercising each of the alternative 
systems and developing MOPs for each alternative. Therefore, a proxy measure was 
developed that could serve as an indicator to the performance of the Define Measures 
function. 

Proxy Definition of PTM: 

This EM was calculated by taking the number of authoritative sources that were 
considered in the process and then dividing by the total number of authoritative sources 
available for the SoS. 

, # Authoritative Sources Considered 

PTM =- 

# Authoritative Sources Available , n 


The concept was that analysts performing the Define Measures function should 
consider all available documentation for the SoS. Considering all available 
documentation helped to ensure that all requirements and testable capabilities are 
captured in the process and subsequently MOPs are defined for each. The team 
considered that in reviewing a wider set of authoritative sources the process would yield a 
higher number of requirements and capabilities, and in turn provide more traceable 
metrics. 
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If an alternative considered every document in the comprehensive list of 
authoritative documents, as discussed in Appendix G, it received a score of 100%. Table 
26, in Appendix G, contains the comprehensive list of authoritative documents, weights 
for each document, and the score for each alternative. If an alternative uses a document 
in its process then the alternative receives the complete score for that document. The 
final results of the PTM EM is included in Table 15. 

2. Results of Quality of Planning Outputs EM 

Quality of Planning Outputs was the EM for the Planning Results (1.4.3) function 
of the JC3M value hierarchy. The objective of the Planning Results function was to 
determine how well an alternative produced planning documents when planning for a C4I 
SoS evaluation. This EM was calculated by assigning an overall quality level to the 
planning documents produced by the alternative. The detailed methodology used to 
derive the Quality of Planning Outputs EM is included in Appendix C. However, the 
results are also summarized here. The results of the Quality of Planning Outputs EM are 
included in Table 16. 

The JC3M team decided to use a Likert scale to determine the quality of planning 
products evaluation measure. The Likert scale was set from 1 to 4 with four being the 
best score and one being the worst score. 

The team assembled a questionnaire (provided at Appendix C) that reviewed the 
quality of planning products. Each question also contained a criterion in order to help the 
respondent differentiate between a best value of 4 or a worst value of 1. 

After creating the questionnaire the team solicited SME input. One SME from 
MCTSSA, and two from China Lake, answered the questionnaire. 

The raw score was calculated by averaging the scores for each alternative from 
the questions answered by each of the SMEs. 
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3. Compiled results for all EMs 

Table 16 depicts the results including both M&S and offline evaluation, which 
provided values to express the performance of the alternatives for every critical EM. 
(Cost is calculated separately.) 



Percentage 

of 

Traceable 

Measures 

Days to 
Plan 

Evaluation 

Quality of 
Planning 
Outputs 

Elasticity of 
Labor 

Elasticity 
of Duration 


(%) 

(Days) 

(1-4 Likert 
Scale) 

(unit less) 

(unit less) 

Ideal Value 

100% 

Less is 
better 

4 is Ideal 

Less is 
better 

Less is 
better 

FEDOS 

0 

140 

3.17 

0.87 

0.87 

MC3T 

72 

121 

3.25 

0.78 

0.78 

JTEM CTM 

92 

73 

3.42 

1.04 

0.83 

SCR 

92 

158 

3.00 

0.98 

0.98 

FCB 

88 

127 

2.75 

0.72 

0.72 


Table 16. Raw Data Matrix. 


This is the Raw Data Matrix with the raw scores for each EM. The results are the raw 
scores that are still in the units of the respective EM. Scores for all alternatives were 
rounded for display in this table. 


G. CALCULATING AN OVERALL UTILITY SCORE 

As stated in Section B “Multi Attribute Utility Theory” of this Chapter, the EMs 
are combined into a single measure for each alternative called the overall utility score. 
Equation 2, the value function equation, was used to calculate the overall utility score. 
Table 17 and Table 18 depict the intermediate and final values obtained using the 
equation. 
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Table 17 depicts the EMs after transforming the raw score into the utility score 
using the Wymorian SSFs. A utility score of 1.0 is ideal. 



Percentage 

of 

Traceable 

Measures 

Days to 
Plan 

Evaluation 

Quality of 
Planning 
Outputs 

Elasticity of 
Labor 

Elasticity 
of Duration 

Range 

0-1 

0-1 

0-1 

0-1 

0-1 

FEDOS 

0.00 

0.66 

0.92 

0.75 

0.75 

MC3T 

0.10 

0.88 

0.94 

0.86 

0.86 

JTEM CTM 

0.98 

1.00 

0.96 

0.42 

0.80 

SCR 

0.98 

0.37 

0.89 

0.54 

0.54 

FCB 

0.90 

0.83 

0.82 

0.92 

0.92 


Table 17. Utility Score Data Matrix. 

This is the Utility Score Data Matrix with the utility scores for each EM. The results are 
the individual utility scores that range from 0 to 1, where 1 is ideal. Scores for all 
alternatives were rounded for display in this table. 

After obtaining the utility scores, the scores are then multiplied by their respective 
weights. Finally, the overall utility score is obtained by adding the individual EM 
weighted utility scores for each alternative. The individual EM weighted utility scores, 
overall utility scores, and LCCE (in millions of dollars) for each alternative are recorded 
in Table 18. 
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Percentage 

of 

Traceable 

Measures 

Days to 
Plan 

Evaluation 

Quality of 
Planning 
Outputs 

Elasticity 
of Labor 

Elasticity of 
Duration 

Overall 

Utility 

(0-1) 

LCCE 

($ Mil) 

FEDOS 

0.00 

0.04 

0.39 

0.06 

0.14 

0.63 

5.01 

MC3T 

0.02 

0.05 

0.39 

0.07 

0.17 

0.71 

5.98 

JTEM CTM 

0.24 

0.06 

0.40 

0.04 

0.15 

0.89 

6.97 

SCR 

0.24 

0.02 

0.37 

0.05 

0.10 

0.79 

7.72 

FCB 

0.22 

0.05 

0.34 

0.08 

0.18 

0.87 

8.13 


Table 18. Quantitative Modeling Decision Matrix. 

This table displays the estimated performance of each alternative as weighted utility 
scores for each EM. The weighted utility scores are summed and displayed adjacent to 
the LCCE (in millions) in the yellow columns. Weighted Utility and Overall Utility 
scores for all alternatives were rounded for display in this table; these rounded values 
were used for Figure 42 and Figure 43. 

H. UTILITY SCORE VERSUS LIFE CYCLE COST ESTIMATE 

The final step of AoA was to compare alternatives side by side using a Utility 
versus Life Cycle Cost Estimate Plot (see Figure 42 and Figure 43). The plot allowed 
decision makers to visually compare the cost of each alternative, the estimated 
performance of each alternative, and the cost to achieve estimated performance in 
specific alternatives simultaneously. This data was utilized in the Final Recommendation 
phase, where a summary of the project results was provided for stakeholder 
consideration. Figure 42 and Figure 43 are similar; Figure 43 is scaled to increase the 
ability to differentiate the relative cost and performance of each alternative. 

FCB has the highest LCCE at $8.13M. SCR is next at S7.72M, followed by 
ITEM CTM which costs $6.97M. FEDOS and MC3T follow with LCCE of $5.01M and 
$5.97M respectively. JTEM CTM and FCB dominate all of the other alternatives for 
utility with scores of 0.89 and 0.87. JTEM CTM and FCB have virtually equal utility, yet 
the JTEM CTM LCCE costs approximately $1.16M less than FCB. JTEM CTM has the 
median LCCE of the five alternatives which, coupled with the highest utility, clearly 
dominated all other alternatives. 
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Utility versus Life Cycle Cost Estimate 
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Figure 42. Utility versus Life Cycle Cost Estimate Plot. 

This plot illustrates the utility of each alternative while displaying the estimated cost to 
achieve that performance. JTEM CTM utility slightly exceeded that of the FCB 
alternative, with scores of 0.89 and 0.87 respectively. JTEM CTM and FCB utility scores 
are followed in descending order by SCR, MC3T, and FEDOS. ECCE values (in 
millions) range from FEDOS at $5.01 to FCB at $8.13. JTEM CTM has the median 
LCCE value at $6.97. 
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Utility 


Utility versus Life Cycle Cost Estimate 



Life Cycle Cost Estimate ($MIL) 


Figure 43. Utility versus Life Cycle Cost Estimate Plot Zoom. 

This plot illustrates the utility of each alternative while displaying the estimated cost to 
achieve that performance. FEDOS had the lowest estimated utility, followed by MC3T, 
SCR, FCB, and JTEM CTM. JTEM CTM and FCB scored 0.89 and 0.87 respectively. 
FEDOS had the lowest estimated cost, followed by MC3T, JTEM CTM, SCR, and FCB. 
JTEM CTM slightly exceeds the other alternatives for utility, and has the median cost of 
the five alternatives. This figure differs from Figure 43 in that the scale is revised, with 
the origin at $4.5M, 0.55 utility in order to more clearly demonstrate the differences 
between alternatives. 
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VII. FINAL RECOMMENDATIONS AND CONCLUSIONS 


A. PERFORMANCE ANALYSIS 

The JC3M team determined the highest performance was exhibited by the JTEM 
CTM alternative, at an overall utility rating of 0.89 on a 1-0 scale. In order of decreasing 
utility, the other alternatives were FCB at 0.87; SCF at 0.79; MC3T at 0.71; and FEDOS 
at 0.63. 

The team’s confidence in the accuracy of the performance evaluations differed 
among the alternatives. The team had very high confidence in the accuracy of the 
performance evaluation of MC3T and FEDOS. The team had less, although still high, 
confidence in the accuracy of the JTEM CTM performance evaluation. The team had 
less confidence in the accuracy of the performance evaluation of FCB and SCR. 

Of all the alternatives considered, the team had the most confidence in the 
estimates of performance calculated for both MC3T and FEDOS. These are the only 
two alternatives that had been utilized in the conduct of full-scale C4I SoS evaluations. 
This real world use of the two alternatives (MC3T and FEDOS) meant the team was able 
to review historical records of actual inputs and outputs of the alternatives, and observe 
the duration of each of these alternatives. Given the number of data sources and recent 
nature of this data, while MC3T was implementing their processes as the JC3M team was 
analyzing alternative solutions, the team had confidence in the accuracy of the estimate of 
performance from these two alternatives. The team has observed first hand the type, 
scope, and duration of efforts involved in MC3T, which are reflected in the overall utility 
score of the alternative, and concluded MC3T will deliver better utility than FEDOS. 

While the JTEM CTM has not been used for full scale C4I SoS evaluations, it is 
the alternative in which the JC3M team had the next greatest confidence in the accuracy 
of the performance evaluation. The JTEM CTM has been reviewed by a very large group 
of stakeholders, the JTEM Community of Interest (COI). These C4I SoS experts have 
provided input to the JTEM CTM methods and processes. A subset of the COI 
participated in the JTEM “Rock Drills” which included a series of tabletop exercises used 
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to evaluate and refine the JTEM CTM. JTEM CTM members have participated in the 
definition and development of JC3M. They have explicitly validated the content and 
definition of the subset of JTEM CTM tasks which were identified as a JC3M alternative 
solution. This validation included a review of the scope, tasks, components, duration, 
and resources required for JTEM CTM to perform a JC3M C4I SoS evaluation. Because 
of the COI review, Rock Drills, and performance validation through modeling of the 
JTEM CTM; the JC3M team has confidence in the accuracy of the estimate of 
performance of this alternative, albeit at a lesser level than the real-world performance of 
both FEDOS and MC3T. 

The team had less confidence in the accuracy of the performance evaluation of the 
SCR and FCB alternatives. Neither of these alternatives benefited from the very 
extensive C4I SoS SME review which supported JTEM CTM. Additionally, neither 
alternative has been used in real world SoS evaluations, unlike FEDOS and MC3T. 

B. COST ANALYSIS 

The JC3M team determined the least expensive alternative was the FEDOS 
alternative, at a life cycle cost of S5.01M over the ten-year projected lifespan of JC3M. 

In order of increasing cost, the other alternatives were MC3T at $5.98M; JTEM CTM at 
S6.97M; SCR at $7.72M; and FCB at $8.13M. 

The team noted that MC3T was estimated to cost approximately $970,000 more 
than FEDOS, which it has replaced, over the ten year span of the LCCE. Because the 
LCCE postulated the conduct of a single C4I SoS evaluation per year through the eight 
year O&S phase, this cost difference equates to approximately $121,000 per C4I SoS 
evaluation. 

The team also noted that FCB is approximately $410,000 more than SCR. 

Unique to FCB is the use of senior SMEs who generate the performance measures, yet 
are not part of the organization conducting the C4I SoS evaluation. The team calculated 
the cost of senior SMEs, and included these as an overall cost to DoD. While the cost of 
their labor was not charged to C4I test organizations, the use of these resources is a cost 
to DoD. If the cost of only personnel organic to the test organization had been reported 
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for FCB, the cost would have been lower by $1.14M but this would have represented 
only part of the cost of DoD resources. 

In a discussion with the Deputy Chief of the Methods and Processes Division 
[Wilson, 2007], the team reviewed the maturity of the JTEM CTM and its readiness for 
implementation. While JTEM plans to deliver two more versions of the CTM by 2009, 
the development between 2007 and 2009 is primarily focused on development of the 
Joint Mission Environment. Wilson [2007] and the JC3M team lead agreed that the CTM 
was robust in its current state, but would require additional development. 

The cost to complete development of the JTEM CTM [Bjorkman (b), 2007], is 
approximately $3.5M through 2009. Although the team included the complete 
development cost of the JTEM CTM in the LCCE, they recognized that development is a 
non-recurring cost that is not borne by any C4I test organization. 

O&S is the most important cost from the perspective of C4I test organizations. 
O&S is a recurring cost, borne by each C4I test organization that implements an 
alternative, for each year that alternative is used. With the exception of the JTEM CTM 
development cost ($3.5M), O&S is the largest portion of the LCCE for each alternative. 
The JTEM CTM alternative had the lowest overall O&S cost at $2.25M over the ten year 
LCCE. This O&S cost was the lowest of all alternatives by $1.66M. 

Of all the alternatives considered, the team had the most confidence in the cost 
estimates of both MC3T and FEDOS. The team was able to review historical records of 
the planned and actual costs of each of these alternatives: the confidence in these cost 
estimates was high. The team had the next highest confidence in the cost of the JTEM 
CTM alternative. The team validated their cost model with the JTEM and included the 
actual funding required to complete the JTEM CTM development in the LCCE. The 
team had less confidence in the cost estimate of the SCR and FCB alternatives. Neither 
of these alternatives have been used in real world SoS evaluations, unlike FEDOS and 
MC3T. Additionally, the models for SCR and FCB are simply estimates and have not 
been validated as in the case of JTEM CTM. 
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C. PERFORMANCE VERSUS COST ANALYSIS 

As stated previously, the JC3M team determined the performance and cost of 
each alternative to be as follows: JTEM CTM at 0.89 and $6.97M, FCB at 0.87 and 
$8.13M; SCR at 0.79 and $7.72M; MC3T at 0.71 and $5.98M; and FEDOS at 0.63 and 
5.01M. 

The team was interested to observe stratification in performance ratings; MC3T 
and FEDOS were relatively close in both cost and utility; FCB and SCR were similarly 
close; and all four of these alternatives can be seen to deliver increased utility in return 
for increased cost. 


The slope of the line expressing the relationship between cost and utility is 
0.070847, where a change in cost results in a change in utility. The equation for 
calculating the slope of the regression line b is given by 



( 8 ) 


where x represents the cost value of a specific alternative, and x the average of the cost 


values, where y represents the cost value of a specific alternative, and y the average of 
the utility values [Baker, 2006: 6]. The regression line (Fi) can be seen in Figure 44. 


Had the team considered alternatives as a portfolio, and combined attributes (the 
C2 SME panel from the FCB alternative, for example, with the MC3T use of Service 
stakeholders) it would have been possible to define mixed-attribute alternatives that 
provided more utility for the same cost. The most cost-effective combinations would 
have formed a curved efficient frontier [Gunser, 2004: 26] of utility for every level of 
cost. Because combinations were not considered, a linear efficient frontier (L 2 ) is defined 
by the three more efficient alternatives and is seen in Figure 44. Given their respective 
utility scores, both the FCB and SCR alternatives are to the right of the frontier, less 
efficient than, and dominated by, the JTEM CTM alternative. 
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Cost vs. Utility 



Figure 44. Cost Versus Utility with Trendline. 

The figure illustrates estimated LCCE and performance values for the five alternatives. 

The LCCE and performance values for four alternatives (FEDOS, MC3T, SCR, and 
FCB) were used to generate a trendline to predict performance based on alternative cost. 

With this trendline, the predicted performance of JTEM CTM is greater than what would 
be expected, based on the JTEM CTM LCCE. The JTEM CTM alternative is above the 
predicted performance value (Ld, indicating it has a higher value to cost ratio than other 
alternatives. An efficient frontier (L 2 ) depicts the relative inefficiency of FCB and SCR 
when compared to JTEM CTM. 

D. CONCLUSION 

The JTEM CTM alternative dominated both FCB and SCR, providing higher 
estimated utility for less cost. JTEM CTM exceeded the performance of all alternatives 
by a very slight margin. Though the team had different levels of confidence in the utility 
scores of each alternative, they are confident that the results and conclusions are valid. 

As described in section A, the team had more confidence in the JTEM CTM utility score 
than in the next highest alternative, FCB. 

The JTEM CTM had the median LCCE, and the lowest O&S cost by a large 
margin. Chapter V section A.2.b “Operations and Support Costs” noted that costs in the 
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O&S phase were primarily driven by the use of personnel resources. The 73 day duration 
of the JTEM CTM planning phase led to the smallest O&S cost. The next alternative, 
MC3T, had a 121 day planning phase, with correspondingly higher O&S costs. This is 
significant because O&S is a recurring cost, borne by every C4I test organization that 
utilizes an alternative. Development of JTEM CTM is the largest portion of the LCCE 
for this alternative. However, the development of JTEM CTM is a nonrecurring cost 
borne by OSD and not borne by any C4I test organization. 

The team recommends monitoring the development of the JTEM CTM for further 
maturation. The JTEM CTM promises significant improvements in the overall utility of 
C41 SoS evaluations, at a significantly reduced operating cost, and deserves further 
investigation. The scope of this project, however, does not allow that investigation. The 
team recommends detailed investigation of the JTEM CTM in its entirety, and optimizing 
of personnel resources and organizations with a modeling tool such as POW-ER. The 
team also recommends for JTEM CTM to conduct a C41 SoS evaluation as soon as 
feasible, to validate the JTEM CTM methods and processes in a “real world” event. 

E. AREAS FOR FURTHER STUDY 

The JC3M team identified several SoS issues in the course of the project, and 
recommended investigation of selected issues when additional time is available. 

1. Establish a Manager 

The team believed the C41 acquisition and testing communities would benefit 
from a dedicated Joint C41 SoS manager. A dedicated SoS manager could provide 
consistency of knowledge in an evolving C41 acquisition and testing environment. Their 
role could include, but should not be limited to, the following tasks: Documenting C4I 
SoS capabilities, long range SoS capabilities planning, testing requirements management, 
reducing SoS testing cycle and costs, assessing the improvements in SoS processes, 
supporting developmental and operational testing as a stakeholder, actively participating 
in IPTs that address SoS testing issues, risk management, and addressing ad hoc SoS 
configuration resulting from new threats and concepts. This is consistent with the 
concepts of capability portfolio management under the Joint capabilities area construct. 
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2. Initiate Risk Management Strategies 

The C4I SoS continues to change configuration as it exhibits new capabilities. 
The continually changing architecture of the C4I SoS increases the probability of 
capability failures. The increasing probability of capability failures creates risks that 
need be managed. The JC3M team believes risk management strategies should be 
developed and applied to the C4I SoS. The JC3M team has compiled a preliminary list 
of risks that should be managed across the C4I SoS, including the lack of a single entity 
responsible for SoS performance; lack of an objective, repeatable, and methodical 
approach to address individual system problems that impact SoS functionality; varying 
levels of maturity of critical systems within the C4I SoS architecture; lack of consistent 
SoS integration of individual systems; and varied interfaces between individual systems 
that comprise the C4I SoS. 

3. Modify the Acquisition Process 

Systems that are components of the C4I SoS have their capabilities defined as if 
they exist in a vacuum, and their impact on C4I SoS capabilities is not considered. 
System level capabilities should be considered in light of their effect on the C4I SoS, 
which is consistent with the concept of capability portfolio management. The DoD C4I 
SoS acquisition process should require component system sponsors to define C4I SoS 
level effects when creating CDDs and CPDs; establish a funding process for SoS testing; 
require systems to identify their effects on the C4I SoS before fielding; include end-to- 
end SoS effectiveness testing as an explicit part of Operational Testing 
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APPENDIX A. PROJECT PLAN 


INTRODUCTION 

This is the Project Management Plan (PMP) for the capstone project to be completed by 
the Marine Corps Tactical Systems Support Activity (MCTSSA) Cohort in the Naval 
Postgraduate School (NPS) Masters of Science in Systems Engineering program. The 
Joint Command, Control, Communications, Computers, and Intelligence (C4I) Capability 
Certification Management (JC3M) project will create a system for certifying the 
capability of a C4I System of Systems (SoS). A System of Systems in the context of this 
project is a group of individual C4I systems, which may not have been designed, 
acquired, or managed as a collective enterprise, but are being put together as such and 
forming an interdependent entity. The JC3M system is intended to perform an 
assessment that will identify the desired war fighting capabilities and ensure that the SoS 
under test meets these requirements in the intended environment. 

PROJECT DESCRIPTION 

Across the Department of Defense (DoD), early C4I systems were designed, acquired, 
and fielded independently, each addressing a single warfighting function, such as 
logistics, fire support, or intelligence. Over time, warfighting has grown in complexity, 
tempo, and scope, so organizations must be able to respond with increased agility across 
greater distances. This complexity is compounded by adaptive and elusive adversaries. 

To combat today’s adversaries, DoD forces fight jointly. 

Individual C4I systems, which may not have been designed, acquired, or managed as a 
collective enterprise, are being put together as such and forming an interdependent entity, 
a SoS. Today’s C4I SoS, whether Joint or Single Service, is required to transport and 
process shared data throughout the operating forces. Problems are abundant because there 
is no baseline, standard configuration, or overall management of the SoS. 

C4I system-level acquisition, testing, and management are well understood, and 
individual systems have performance requirements. However, ever-changing 
configurations of C4I SoS may not have formally established performance requirements, 
nor threshold values that can be used to evaluate performance. There is not a clear 
understanding of how to manage or assess C4I SoS performance or C4I SoS capability to 
support Joint or single Service missions. A C4I SoS-level capability is a task achievable 
by multiple enterprise components that is not achievable by a single enterprise 
component working on its own. Examples of C4I SoS capabilities include Call for Fire, 
Immediate Close Air Support, and Building a Co mm on Operational Picture. Processes 
and methods for designing and executing C4I system tests are well defined and executed, 
but testing at the SoS level is not well defined, nor are consistent standards and practices 
applied. A complicating factor is that real instances of the C4I SoS have a practically 
infinite number of possible configurations. 

The Joint Interoperability Test Command (JITC) tests the interoperability of systems, but 
this proves that system interfaces function. There is no agency that assesses the 
capability of a SoS to accomplish a task that requires the coordinated, successful 
integration of functions and interfaces across multiple systems. The Marine Corps, for 
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example, has extensive doctrine for the conduct of artillery fire support, but there are not 
documented, testable values which can be used to assess the success of a fire mission. If 
a Forward Observer (FO) had to initiate a Call For Fire five times, did the C4I SoS 
demonstrate a successful capability to provide fire support? What if the FO had to try 
seven times? Three times? What constitutes success? This lack of a consistent SoS 
performance requirement process bedevils all of DoD. 

JC3M system is important because it provides the acquisition community much-needed 
performance requirements for the design of new systems, integration of legacy systems, 
and SoS testing. JC3M system is also important because it provides system integrators a 
tool to assess integration formally, it documents system capabilities and construction, and 
it provides confidence to the warfighter that the C4I SoS works. Every C4I SoS has been 
custom built to date, with all the Configuration Management (CM), troubleshooting, 
training, and support challenges this “one-off’ approach implies. With a consistent 
assessment methodology and a documented baseline configuration, C4I SoS support 
becomes repeatable, and CM, training, troubleshooting, and knowledge management 
costs shrink. 

The Joint Test Evaluation Methodology (JTEM) team is addressing Joint SoS 
interoperability from the Office of Secretary of Defense (OSD) level. Marine Corps 
Systems Command (MARCORSYSCOM), the acquisition organization for the Marine 
Corps, is approaching the issue from a Service perspective. MARCORSYSCOM has 
tasked MCTSSA to develop Marine Air Ground Task Force (MAGTF) C4I Capability 
Certification Management (MC3T), a methodology for managing the MAGTF C4I SoS 
as a single system, in accord with modem systems engineering practices. MC3T will 
manage the MAGTF C4I SoS as a set of SoS-level capabilities, rather than as a fixed 
hardware or software baseline. 

The JC3M project team will define a system that will assist a test agency in performing a 
C4I SoS assessment that will identify the desired war fighting capabilities and ensure that 
the system under test meets these requirements. The JC3M system will include Doctrine, 
Organization, Training, Materiel, Leadership, Personnel, and Facilities (DOTMLPF) 
considerations in designing the system to be used by an organization, following 
repeatable processes, and consistent resources. The processes will not only be usable by 
MC3T, but can, with appropriate Service-specific modifications, be utilized by other 
Services and Joint agencies. 

ORGANIZATION 

The JC3M project team organization is provided in Figure 45. 

JC3M 

POINT OF CONTACT 
\___/ 


EDITOR 
IN CHIEF 


LIBRARIAN AND 


BUSINESS AND 


MODELING AND 



PROJECT 

RESEARCH j 


COST 


SIMULATION , 



SCHEDULE 


Figure 45. JC3M Team Organization. 
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Roles & Responsibilities 

The JC3M project team consists of 14 interdisciplinary team members located in 5 
different geographical locations. Table 19 is a listing of each team member’s roles and 
responsibilities. Roles and responsibilities are subject to change as the project progresses. 


Name 

Roles & Responsibilities 

Location 

Lamb, Jeremy 

Librarian & Researcher 

Bethesda 

Martin, Calvin 

Meeting Minutes, Librarian & Researcher 

China Lake 

Acosta, Jacobo 

Business &Cost 

Corona 

Huseth, Scott 

Meeting Minutes, Business & Cost 

Corona 

Medina, Vince 

Business & Cost, Modeling & Simulation 

Corona 

Trinh, Khai 

Modeling & Simulation 

Corona 

Hoesly, Scott 

Meeting Minutes, Business & Cost 

MCTSSA 

Medina, Jorge 

Business & Cost 

MCTSSA 

Nguyen, Michael 

Librarian & Researcher 

MCTSSA 

Patel, Jay 

Project Schedule, Librarian & Researcher 

MCTSSA 

Rangi, Kamaljit 

Librarian & Researcher, Modeling & Simulation, Project Schedule 

MCTSSA 

Schoen, Tim 

Librarian & Researcher, Modeling & Simulation 

MCTSSA 

Willis, Ron 

POC, Business & Cost 

MCTSSA 

Krider, Steven 

Editor in Chief 

Philadelphia 


Table 19. Team Member Listing. 

Project Advisors 

The JC3M project team advisor is Gregory Miller. The second reader is Professor David 
Hart. Both are NPS faculty members in the Department of Systems Engineering. 

APPROACH FOR PMP UPDATES 

The JC3M project team will review the PMP throughout the NPS Capstone project. If 
significant discrepancies or errors are found during a review, the PMP will be updated. 
The Editor in Chief will incorporate the changes and submit the revised PMP to the 
Capstone Advisor for review and approval. If a change in scope or engineering approach 
induced the revision, the JC3M project team will resubmit the PMP to the Systems 
Engineering Department Chair for approval. 

CONFIGURATION MANAGEMENT 

The deliverables created by the JC3M project are in the form of documents, 
presentations, and simulation results. The Editor in Chief will maintain a copy of all 
deliverables and deliverable revisions in a chronological archive. The JC3M project team 
will revise each of the deliverables prior to being finalized for submittal. This JC3M 
project team edits will be tracked utilizing a system of incremental alphanumeric 
revisions (i.e. PMP Rev A to PMP Rev B). Once a deliverable is ready for submittal it 
will be published utilizing numeric revision (i.e. PMP Rev 1). 

TECHNICAL REVIEWS 
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The JC3M project team will employ technical reviews as a way to ensure accuracy, 
validity, and appropriateness of progress. Work products will be reviewed internally and 
by stakeholders to ensure all parties understand the problem, the approach, and the 
solution. Technical reviews precede formal control gates, set expectations for all 
stakeholders, and reduce the number of surprises at formal reviews. 


SYSTEMS ENGINEERING APPROACH 


OVERVIEW 


The JC3M project team will implement a Systems Engineering approach that starts with 
the identification of customers’ needs and proceeds through the phases illustrated in 
Figure 46 until a recommended solution is generated. Each phase presents the 
opportunity to re-evaluate progress and return to a prior stage for refinements if 
necessary. This will be critical to adapting the project to ensure the customers’ and 
stakeholders’ needs are being achieved. The SE process model below is based on 
combination of System Engineering Design Process (SEDP) by Prof Paulo and 
SIMILAR System Engineering Process Model from INCOSE, modified to fit this project. 



Figure 46. JC3M Systems Engineering Process. 

JC3M Systems Engineering Process Phases 

Each of the JC3M Systems Engineering Process phases is explained herein. 

Problem Refinement Phase 

The JC3M project team will utilize this phase to clarify the customer’s needs and begin 
managing the JC3M project risks. The outputs of this phase will allow the JC3M project 
team to design multiple solutions for the JC3M system problem. The Problem 
Refinement Phase is composed of four key elements described below. 
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Element 1: Needs Analysis 

This element will provide justification for proceeding further in the design process. It 
will perform System Decomposition in order to gain an initial understanding of the 
process in question and to begin to look at the process in terms of how it fits into the “big 
picture”, and Functional Analysis in order to identify and decompose the critical 
function(s). It will take primitive needs elicited during stakeholder analysis and 
transforming them into effective needs. If conflicts occur, compromise will be sought 
after when possible, however, key stakeholders such as JTEM and the NPS Systems 
Engineering Department will have stronger influence than others. 

Inputs: Initial Problem Statement, Stakeholders Needs/Wants 

Tools: Functional flow diagrams 

Outputs: Revised Problem Statement 

Element 2: Requirement Generation and Analysis 

During this element, the JC3M project team will generate a set of processes that reflect 
stakeholders’ needs. Requirements analysis will assists the customers in refining their 
requirements in concert with defining functional requirements. Stakeholder Analysis will 
also be performed to identify people and organizations relevant to our problem and to 
determine their needs, wants and desires. Stakeholders will have a vested interest, or 
personal stake, in our problem and/or its eventual solution. 

Inputs: Revised Problems Statement 

Outputs: Initial Problem Refinement Report (PRR), Functional Hierarchy 
Element 3: Value System Design 

The JC3M project team will use this element to help establish processes, objectives, and 
evaluation measures. Value System Design will establish a qualitative value hierarchy 
tree that identifies processes, objectives and sub-objectives, and evaluation criteria. 

Value System Design will identify system characteristics that reveal measurable 
parameters. These parameters will identify the metrics for evaluation of alternative 
solutions. Measures of Effectiveness (MOE) and Measures of Performance (MOP) will 
be generated from the value hierarchy tree. MOEs will reflect operational objectives and 
be understandable to decision-makers. MOPs will provide measurable results that can 
demonstrate progress towards goals and objectives, provide specific measurement results, 
and determine effectiveness. MOPs will determine if the system is meeting its mission, 
vision, and goals. 

Inputs: Initial PRR 

Tools: Value Hierarchy Tree 

Outputs: MOEs, MOPs 
Element 4: Initiate Risk Management 

The JC3M project team shall Initiate Risk Management by identifying potential 
opportunities for risks, assessing their associated probabilities of occurrence, and 
determining their impact to the project. Risk management will address: 

Requirements risk - The JC3M project team will identify and describe the requirements 
of JC3M system with stakeholders, and aggressively manage the scope of requirements to 
minimize scope creep. 

External risks - Is the process development dependent on external events or 
accomplishments over which the program has no control? 
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Development risk - Can the process be designed so that required function and 
performance are met within constraints? 

Resource availability - Are required personnel and facilities available? 

Resource risks - Are funding, training, schedule (time) and tools available? 

The JC3M project team will manage these risks by developing and documenting a risk 
management strategy. This strategy will include risk planning, assessment, handling, and 
monitoring. For each identified risk item, probability/consequence scales will be 
developed and ratings will be assigned. Supporting analysis will help rate, prioritize, and 
aggregate risks in order to minimize potential areas of concern. This analysis will be 
conducted in accordance with the Risk Management Guide For DoD Acquisition. 

Design Alternatives Phase 

The JC3M project team will utilize the Design Alternatives Phase to generate multiple 
solutions to the problem, establish feasibility criteria and apply those criteria to eliminate 
those alternatives that are clearly infeasible. The creation of these solutions requires the 
outputs of the Problem Refinement Phase. The outputs of this phase will be used for the 
creation of models in the Model the Alternatives Phase and for the Analysis of 
Alternative (AoA) portion of the Simulation and Analysis of Alternatives Phase. The 
Design Alternatives Phase is composed of three key elements. 

Element 1: Alternative Generation 

In this element the JC3M project team will prioritize the critical functions and sub¬ 
functions that the system under design must be able to perform. These critical functions 
will be a subset of the functions defined in the Value Hierarchy during Value Systems 
Design. Next, the JC3M project team will brainstorm and research alternative ways to 
perform the critical system functions. Finally, the JC3M project team will build 
alternative packages that account for each function identified. Zwicky’s Morphological 
Box (ZMB) will be used as a tool to develop the alternative packages. The output of this 
element will be a set of alternative solutions for the system under development. 

Inputs: Value Hierarchy Tree, MOEs, MOPs, and PRR 

Tools: ZMB, brainstorming, and research 

Outputs: Set of alternatives 

Element 2: Feasibility Screening 

The purpose of this element is to eliminate from further consideration those alternatives 
that are clearly infeasible so as not to waste valuable effort in the Model the Alternatives 
Phase. First, screening constraints will be defined. Most of the screening constraints will 
come directly from the constraints on the system detailed in the Value System Design 
element of the Problem Refinement Phase. The JC3M project team will also obtain 
screening constraints from the stakeholders. 

Those alternatives that meet all of the system constraints are considered feasible, while 
those that clearly fail to meet one or more constraints are considered infeasible. The 
screening process will only screen out those alternatives that are clearly infeasible. The 
JC3M project team will use a Feasibility Screening Matrix (FSM) to organize the results 
and identify the feasible alternatives. 

Inputs: Set of alternatives, SRD, MOEs, MOPs, and Stakeholder input 

Tools: FSM 

Outputs: Feasible alternatives 
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Element 3: Alternative Scoring Criteria 

The purpose of this element is to define the criteria for scoring alternatives based on 
MOEs and MOPs. Stakeholders must agree on the values that will be later used to score 
the final alternatives. Numbers quantify values and uncertainties; humans can 
understand, compare, and manipulate numbers. These criteria will be generated early, so 
stakeholders can review and validate them. Alternative scoring criteria provide 
quantitative support for the decision-makers and enables consistent evaluation of 
alternatives. The model that we will use is the quantitative value model. Quantitative 
value models are the scoring functions and weights that will be used during the Analysis 
of Alternatives to evaluate and compare our alternatives. 

Input: Feasible alternatives, MOEs, MOPs, Value Hierarchy Tree 

Output: Scoring Criteria, Preliminary Quantitative Value Modeling Decision Matrix 

Model the Alternatives Phase 

The JC3M project team will utilize this phase to generate models based on the 
alternatives selected in the Design Alternatives Phase. The outputs of this phase are the 
models of the alternatives that will be simulated during the Simulation and Analysis of 
Alternatives Phase. The Model the Alternatives Phase is composed of the two key 
elements described below. 

Element 1: Model Development 

Models will be developed based on the alternatives selected in the Design Alternatives 
phase in accordance with Department of Defense Architecture Framework (DoDAF), 
MAGTF, and Joint Services architectures. SoS interoperability diagrams based on legacy, 
current, and future systems will be utilized to create detailed functional and behavioral 
models. 

Inputs: Feasible alternatives, DoDAF, Joint Service architectures, and Functional 

Hierarchy 

Tools: Microsoft Visio and Vitech COREsim 

Outputs: Detailed functional and behavioral models 
Element 2: Cost Analysis 

Based on the system requirements, the system life cycle, and activities in each phase of 
the life cycle, the Cost Breakdown Structure (CBS) will be developed and a life cycle 
cost estimate for each alternative will be developed. This will include a business model 
for the activity performing the test functions detailed in the final process. 

Inputs: Detailed functional and behavioral models, existing cost data (stakeholders 

and others will be consulted for existing cost data) 

Tools: Microsoft Excel and Operating and Support Cost Analysis Model (OSCAM) 

Outputs: SoS Test Process Cost Estimates 

Simulation and Analysis of Alternatives Phase 

The JC3M project team will utilize the models generated earlier to evaluate and rank each 
of the alternatives. The output of this phase will provide a recommended solution to be 
published in the Final Recommendation Phase. The Simulation and Analysis of 
Alternatives Phase is composed of the two key elements described below. 

Element 1: Simulation 
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The simulation model will be probabilistic, with elements of uncertainty as defined in the 
Problem Refinement Phase. The results from the simulation will provide a means of 
predicting or estimating the performance of our design alternatives with respect to the 
selected evaluation measures. 

Input: Detailed functional and behavioral models 

Tools: Vitech COREsim, SimVision and Arena 

Outputs: Simulation results 
Element 2: Sensitivity Analysis 

The JC3M project team will utilize sensitivity analysis to determine the factors that 
contribute the most to variability in the simulation outputs. 

Input: Simulation results 

Outputs: Factors that contribute to the output variability 
Element 3: Trade off Studies 

Trade-off studies for the models and will be based on cost, schedule, and performance of 
each alternative. 

Input: Simulation results 

Outputs: Preliminary alternative ranking 

Element 4: Analysis of Alternatives 

This phase compares the alternative models, after their simulation performance, using 
MOEs, MOPs, and Quantitative Value Modeling Decision Matrix, to rank the 
alternatives. Once the structure and numbers are in place, and the stakeholders agree, the 
analysis can begin. 

Inputs: Simulation results, factors that contribute to the output variability, scoring 

criteria, Preliminary Quantitative Value Modeling Decision Matrix, Preliminary 
alternative ranking 

Tools: Quantitative Value Modeling Decision Matrix 

Outputs: Ranked Alternatives 

Final Recommendation Phase 

The JC3M project team will assemble all earlier inputs into a cohesive recommendation 
for implementation, and publish this as a final report. The output of the Final 
Recommendation Phase will be provided to both the ITEM and MC3T teams for their 
use. The MC3T implementation team is a critical stakeholder, but their schedule includes 
a proof of concept event in October 2007. Based on this schedule, the MC3T team 
concluded they cannot wait for JC3M project outputs. MC3T will instead implement an 
interim process, and will review the JC3M project report for inclusion of applicable 
recommendations as MC3T refines their processes. The Final Recommendation Phase is 
composed of one key element described below. 

Element 1: Publish Recommendation 

In this phase, all outputs from earlier phases will be assembled into a final report. This 
report will be briefed and provided to stakeholders at the end of the project. 

Inputs: Ranked Alternatives 

Outputs: Final Report describing a recommended approach for the conduct of C4I SoS 
capability testing 
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Assess Performance Phase 

This project will not completely conclude after delivery of the final report. Some 
members of the JC3M project team will continue to work with MC3T, ITEM, and other 
stakeholders after this project concludes. Some members of the team already participate 
in the JTEM Capability Testing Community of Interest, where they will share their 
experiences with other members and continue to advance the art of capability testing. 

STAKEHOLDERS 

Identification of stakeholders - organizations and personnel with an interest in, and some 
authority over, the final content of the project - is a critical task. Too few stakeholders 
can result in an incomplete problem assessment, with a resulting solution that is narrow. 
Too many stakeholders, with varying interests, can dilute the focus of the project, or 
increase the scope beyond the ability of the project team to address within a limited time. 
The project team will review the project history and description to identify stakeholders. 
The team will interview the stakeholders to validate their interest and authority, as well as 
to identify possible additional stakeholders. 

The stakeholders are: 

• NPS Systems Engineering Department will validate the appropriate problem and 
solution approach. 

• JTEM team, Joint Test and Evaluation Project, is tasked to develop, test, and 
validate a methodology for defining, developing, and using a test environment to 
support test and evaluation of system performance in a Joint mission operational 
environment. JTEM reports through the Deputy Director, Air Warfare 
Operational Test and Evaluation, to Director, Operational Test and Evaluation 
(DOT&E) to the OSD. The JTEM lead is very interested in how both the JC3M 
project and MC3T solve some of the same challenges JTEM faces. 

• MC3T team at MCTSSA, which may use the project output to modify their C4I 
SoS testing processes. 

• Marine Corps Combat Development Command (MCCDC) is the agency that 
defines doctrine and requirements for the Marine Corps. MCCDC will be 
engaged in defining MAGTF C4I SoS requirements, and will be included in 
MC3T processes. 

Possible additional stakeholders: 
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• Central Technical Support Facility (CTSF), Ft. Hood, TX, which performs many 
SoS tests for the U.S. Army, like MCTSSA, and may be interested in SoS test 
requirements generation and conduct. 

• U.S. Army Test and Evaluation Command (ATEC), Alexandria, VA, plans, 
conducts, and integrates developmental testing, independent operational testing, 
independent evaluations, assessments, and experiments. 

• U.S. Navy Operational Test and Evaluation Force (OPTEVFOR), Norfolk, VA, 
assesses the operational effectiveness and suitability of new and improved war 
fighting systems and capabilities for the Navy. 

• JITC has the mission to provide operational test and evaluation of C4I systems, 
certify C4I systems interoperability and solve interoperability deficiencies. 

• Air Force Operational Test and Evaluation Center, Kirtland Air Force Base, NM, 
assesses the capability of new systems to meet warfighter needs by planning, 
executing and reporting independent operational evaluations. 

REQUIREMENTS 

At this stage of the project, the JC3M project team knows that the system to be designed 
must be relatively affordable, flexible enough to work with a variety of SoS 
configurations, and relatively quick to execute. As defined in the process approach, 
refinement of requirements, as well as the possible discovery of new requirements, may 
affect a balanced, life-cycle solution. Additional requirements will be identified during 
the needs analysis element of the Problem Refinement Phase by conducting system 
decomposition, stakeholder analysis, and functional analysis, These requirements will be 
documented in the SRD that will be reviewed and approved by the stakeholders prior to 
the Design Alternatives phase. 

TOOLS 

The JC3M project team plans to utilize the tools identified in each phase and element 
during the execution of the project. However, the JC3M project team may determine that 
the identified tools are inadequate, unnecessary, or undesirable. If this occurs new tools 
will be researched and selected, if required, to complete the project elements. 

RISK MANAGEMENT 

Risks to the project will be defined and managed by the business and cost sub-team. 

Risks will be identified during project analysis, confirmed with primary stakeholders, and 
continually managed. Risks will be managed by prioritization and recording, creation of 
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a risk management plan for each significant risk, and ongoing reporting of risks and 
associated issues. Initial risks are identified below, along with their mitigation approach. 
Personnel risk is medium. Student teams will be assigned to each task. This project 
should be the focus of the student’s efforts, which reduces the risk of personnel 
reassignment. This risk will be reduced by ongoing communication between project sub¬ 
teams. Personnel outside the student teams (references, authorities, and contributors) 
may be reassigned, unavailable, or slow to respond, and thus present a medium risk. This 
risk will be managed by ongoing communication with outside personnel, as well as 
communications with stakeholders on the progress of the project, or schedule slips. 

Time risk is medium. The project must be completed by September 2007, which strictly 
limits this resource. If the project grows from the current scope, there may not be 
sufficient time to complete the project. This risk will be managed by aggressively 
monitoring the scope of the project, communicating with stakeholders, and managing 
scheduled activities. 

Lack of resources is a low-medium risk. The primary equipment need identified to date 
is a simulation tool for modeling processes. The use of these tools introduces some risk 
(availability, speed to learn, suitability), but this will be managed by utilizing simple, 
accessible, and suitable tools where appropriate. Because this is a student-run (unfunded) 
project, there is a medium risk of not having suitable simulation tools due to lack of 
funds. If funds are lacking, no- or low-cost, non-simulation-specific tools will be 
identified for use. 
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MI LESTONES & DELIVERABLES 

Table 20 lists each milestone and Interim Progress Review (IPR) associated with this 
project. Each milestone will have at least one scheduled meeting with the stakeholders to 


discuss decisions and deliverables for that milestone. The stakeholders and the JC3M 
project team must agree that the required deliverable(s) are completed satisfactorily 
before a milestone is considered complete. __ 


Milestone 

IPR 

Description 

Deliverable 

Date 

1 

- 

PMP Approval 

Project Management Plan 

16 Feb 2007 

2 

1 

Requirements 

Approval 

System Requirements 
Document 

16 Mar 2007 

3 

2 

Alternative Scoring 
Matrix Approval 

Alternative Matrix 

18 Apr 2007 

4 

3 

Models of Alternatives 

Conceptual Alternative 
Models & Descriptions 

21 Jun 2007 

5 

4 

Alternative Selection 

Alternative Selection 
Report 

16 Aug 2007 

6 

5 

Report Delivery & 
Project Presentation 

Project Presentation and 
Final Report 

14 Sep 2007 


Table 20. List of Milestones and Deliverables. 
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SC H EDULE 


The schedule for the JC3M project team is provided in Figure 47 
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APPENDIX B. NEEDS ANALYSIS QUESTIONNAIRE 


After the definition of the primitive need, requirements were captured via 
stakeholder focused interviews, aided by questionnaires. The multiple responses to the 
first questionnaire were received with mixed results. All of the responses have not been 
incorporated due to lack of quality and importance. The questionnaires were reviewed 
verbally with the stakeholder or provided to them for review electronically; responses 
from individuals and organizations follow the questionnaire. After reviewing the 
responses from the first questionnaire the team decided to adapt their approach and 
generate a second questionnaire with more pertinent questions. JITC’s responses from the 
second questionnaire are provided. 

QUESTIONNAIRE VERSION 1. 

SYSTEM OF SYSTEM PERFORMANCE ASSESSMENT 

As a Systems Engineering project for the Naval Postgraduate School, Monterey, 
CA, students are creating a system for certifying that a Command, Control, 
Communications, Computers, and Intelligence (C4I) System of Systems (SoS) can 
provide stated capabilities. Students are contacting organizations in order to understand 
current practices, establish a performance baseline, and identify alternative approaches to 
a solution. 

Your organization, which designs, creates, or maintains a complex SoS, has been 
identified as a candidate to answer this questionnaire and participate in focused 
interviews, An example SoS would be a commercial passenger aircraft developed as a 
single system, which incorporates components developed for other aircraft, rather than 
developing new components. The SoS would be the airframe, avionics, engines, radar, 
etc. that make up the entire aircraft capability. Your support is requested because it will 
directly assist the student team, advance the art of Systems Engineering, and increase the 
efficiency and effectiveness of C4I SoS testing. 

1. SoS Definition: 

a. What is your SoS? 

b. Where is it used? 

c. What does it do? 

d. What are the high-level components of your SoS? 
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e. Are components created or managed independently, or are they managed 
centrally? 

f. Was your SoS built to a systems architecture model, as an automobile 
might be, or did it evolve, as the Internet has? 

2. Requirements: 

a. Who creates requirements for your SoS? 

b. Are requirements generated within or outside your organization? 

c. Does the requirements generator have any authority over construction, 
operation, or maintenance of the SoS? 

d. Do requirements for the SoS change over time? 

e. Are requirements documented? 

f. Can I review your requirements? 

3. Validation and Verification (V&V): 

a. Is validation (“does it do the right things?”) or verification (“does it 
perform to specifications?”) required for your SoS, or for component 
systems? 

b. Who performs V&V for your SoS? Is this an internal or external 
organization? 

c. What are sources of V&V requirements: are they derived from system 
requirements, user requirements, or other sources? 

d. What are the consequences of V&V failure? 

e. How is V&V performed for your SoS? 

f. If components of the SoS change, is testing performed? 

g. Can I view your test procedures and results reports? 

h. Is there a process to define: 

i. test environment, 

ii. test criteria, 

iii. test procedures, 

iv. test conduct, and 

v. results reporting? 

4. Support: 

a. Is your SoS supported by an internal or external organization? 

b. How is your SoS maintained? 

c. Is maintenance performed by an internal or external organization? 

d. Is the SoS subject to change? 

e. How is the configuration of the SoS initially recorded, how is it checked, 
and how is it controlled? 

f. If part of the SoS is inoperative, how do users request service? 

g. How are users trained on functions of the SoS? 

h. Are there user, support, or other training resources I can view? 

i. How are support personnel trained? 
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5. SoS Operations: 

a. What is the life cycle of your SoS? 

b. Does your SoS operate continuously or intermittently? 

c. How long do you expect your SoS to function? 

d. Does your SoS have documentation? 

e. Can I view the documentation? 

f. Is there an alternative or backup to your SoS? 

g. What are the consequences (risk of injury or loss of life, risk of damage or 
loss of equipment, risk of financial loss, other risk) of SoS failure? 

6. Your Organization: 

a. How long has your organization been in operation? 

b. Does your organization follow INCOSE, IEEE, ANSI or other process 
standards? 

c. How long have you operated (sold, created, supported) your SoS? 

d. Is there anything you would like to add about your organization or SoS? 


MC3T Response to Questionnaire Version 1 

1. SoS Definition: 

a. What is your SoS? Marine Corps Systems Command 

(MARCORSYSCOM) is developing a process to manage the certification 
of the capability of Command, Control, Communications, Computers, and 
Intelligence (C4I) Systems of Systems, used by a Marine Air Ground Task 
Force, to perform their required functions. The process is called MAGTF 
C4I Capability Certification Test - MC3T. 

b. Where is it used? MC3T will be a distributed operations, used at Marine 
Corps Tactical Systems Support Activity (MCTSSA), Camp Pendleton, CA; 
Marine Corps Systems Command, Quantico, VA; and other sites. 

MCTSSA performs C4I systems interoperability testing and support, as 
tasked by the Deputy Commander for C4I Integration (C4I/I) at 
MARCORSYSCOM. 

c. What does it do? MC3T will assist in managing the MAGTF C4I SoS as a 
single system, in terms of enterprise capabilities. With that definition, 
MC3T will test to ensure the SoS performs to requirements; MC3T will 
also certify for the Operating Forces that if the SoS is configured in a 
defined manner, it will meet performance requirements. In the former 
role, MC3Tassists acquisition managers to ensure their systems support 
SoS level capabilities; in the latter role, MC3T assist the operating forces 
use the SoS effectively. 

d. What are the high-level components of your SoS? MC3Tfunctional 
components consist of requirements definition, capability definition, 
certification, and documentation. 
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e. Are components created or managed independently, or are they managed 
centrally? MC3Tfunctional components are managed independently. 

i. Marine Corps Combat Development Command (MCCDC) 
creates doctrine (how components are used) and requirements 
(what components must do) for the Marine Corps. 

ii. MARCORSYSCOM Program Managers acquire components, 
while a separate MARCORSYSCOM agency (C4I/I) manages 
SoS integration. 

iii. Marine Corps Operational Test and Evaluation Activity 
(MCOTEA) performs Operational Tests; program managers 
perform development testing. 

f. Was your SoS built to a systems architecture model, as an automobile 
might be, or did it evolve, as the Internet has? The MAGTF C4I SoS has 
evolved to its current state. MC3T will be built to a systems architecture. 

2. Requirements: 

a. Who creates requirements for your SoS? 

i. MCCDC defines the requirements for systems, but not the SoS; 

ii. MARCORSYSCOM Program Managers acquire systems which 
must meet requirements; 

iii. MARCORSYSCOM C4I/I determines C4I integration 
requirements, and leads MC3T development with MCTSSA 
support. MCCDC will work with C4I/I and MCTSSA to define 
enterpriseOlevel requirements. 

iv. MCOTEA defines how systems are tested, and conducts or 
coordinates operational tests only. 

b. Are requirements generated within or outside your organization? See 
above. 

c. Does the requirements generator have any authority over construction, 
operation, or maintenance of the SoS? 

i. No. MCCDC has no responsibility for the construction, operation, 
or maintenance of the SoS. 

ii. MCOTEA has no responsibility for the construction, operation, or 
maintenance of the SoS. 

iii. MARCORSYSCOM Program Managers have no responsibility for 
the construction, operation, or maintenance of the SoS. 

iv. MARCORSYSCOM C4I/I has no responsibility for the acquisition 
of the SoS, but is responsible for the integration of the SoS 
components. 

d. Do requirements for the SoS change over time? Yes 

e. Are requirements documented? SoS requirements are documented a 
component (C4I system) level, but not at the SoS level. One MC3T goal is 
to define SoS level requirements. 

f. Can I review your requirements? As requirements are generated they will 
be made available. 
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3. Validation and Verification (V&V): 

a. Is validation (“does it do the right things?”) or verification (“does it 
perform to specifications?”) required for your SoS, or for component 
systems? 

i. No. The MAGTF C4I SoS exists, and has not been required to 
have validation or verification performed. 

ii. Components of the SoS that are a Program of Record do have 
V& Vperformed, to ensure they meet their system level 
requirements. 

iii. MARCORSYSCOM C4I/I chartered Federation Of Systems 
(FEDOS) testing in the past. FEDOS defined a limited C4I SoS, 
with a controlled configuration of hardware and software. 

b. Who performs V&V for your SoS? Is this an internal or external 
organization? 

• MCTSSA conducted FEDOS for MARCORSYSCOM C4I/I. 

• MCTSSA will conduct MC3 T testing for MARCORSYSCOM 
C4I/I. 

c. What are sources of V&V requirements: are they derived from system 
requirements, user requirements, or other sources? 

• For the conduct of FEDOS and predecessor tests, MCTSSA 
conducted a search for SoS V& V requirements. There are no 
formal SoS-level requirements at all. Where available, V& V 
requirements were determined, in descending preference, from: 

o Integrated Architecture Data Stores 
o Doctrine 

o Training and Readiness Manuals 
o Schoolhouse Documents 
o System Manuals & Help Files 
o Unit-level Standard Operating Procedures (SOP) 
o Subject-matter Experts 
o Best guess synthesis 

d. What are the consequences of V&V failure? 

• Prior to MC3T, failure of V& V, for a system in acquisition or 
sustainment, was communicated to the “owning” PM. Because 
SoS level V& V was not a requirement, the PM could ignore the 
failure, address the failure with a risk mitigation strategy, or 
rebuild/refine the system. 

• The consequences of failure of MC3T, i.e. failing to 
demonstrate the defined capability, are not defined. 

e. How is V&V performed for your SoS? MC3T is developing the capability 
certification testing process. 

f. If components of the SoS change, is testing performed? MC3T will 
incorporate capability testing for new or modified components of the 
MAGTF C4I SoS. 
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g. Can I view your test procedures and results reports? MC3T will conduct 
the first test event in October 2007. Procedures will be made available for 
review as they are developed. Test methodology documents are more 
important. 

h. Is there a process to define: 

i. test environment, 

ii. test criteria, 

iii. test procedures, 

iv. test conduct, and 

v. results reporting? MC3T will conduct the first test event in 

October 2007. Definitions, procedures, and criteria will be made 

available for review as they are developed. 

4. Support: 

a. Is your SoS supported by an internal or external organization? The 
MAGTF C4I SoS is supported by: 

• MCTSSA Operating Forces Tactical Systems Support Center 
(OFTSSC) provides 24x7 remote support for tactical C4I 
systems to the Marine Corps and other Operating Forces. 

• Marine Corps Network Operations and Security Command 
(MCNOSC) provides network defense and technical support. 

• Marine Corps Operating Forces operate tactical C4I systems 
around the world. 

• MARCORSYSCOM Program Managers provide varied levels 
of support for their systems. 

b. How is your SoS maintained? By the Operating Forces, MCNOSC, and 
MARCORSYSCOM. 

c. Is maintenance performed by an internal or external organization? 
Maintenance is performed by MCNOSC and the Operating Forces; both 
are external to MCTSSA. 

d. Is the SoS subject to change? Frequently. 

e. How is the configuration of the SoS initially recorded, how is it checked, 
and how is it controlled? This may be an operational question, and 
requires vetting. See Annex K to the Oplan. 

f. If part of the SoS is inoperative, how do users request service? Direct 
contact with MCNOSC or Operating Forces Tactical Systems Support 
Center. 

g. How are users trained on functions of the SoS? Users may receive 
training on component systems. There is no SoS level training. 

h. Are there user, support, or other training resources I can view? Resources 
can be made available on a case-by-case basis. 

i. How are support personnel trained? SoS support personnel at the OFTSSC 
are component (system-level) experts, with formal training on a system. 
They are cross trained on other systems over time. 
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5. SoS Operations: 

a. What is the life cycle of your SoS? 

• Once the MAGTF C4I SoS is established, it remains 
operational until a) the MAGTF is disbanded and returns to 
garrison, or b) the Marine Corps operation (Combat, 
Humanitarian Assistance, Training Exercise) is completed. 

• MC3T is being defined, with the first demonstration of a test 
event in October 2007. 

b. Does your SoS operate continuously or intermittently? 

• The MAGTF C4I SoS operates as described above. 

• MC3T is not an SoS, but will operate on a continuous basis, 
conducting assessments as needed. 

c. How long do you expect your SoS to function? For the foreseeable future. 

d. Does your SoS have documentation? MC3TIs not an SoS, but 
documentation is being developed. 

e. Can I view the documentation? As it is developed. 

f. Is there an alternative or backup to your SoS? The current ad-hoc system 
is the alternative. 

g. What are the consequences (risk of injury or loss of life, risk of damage or 
loss of equipment, risk of financial loss, other risk) of SoS failure? 

• IfMC3T works, it will reduce systems development, 
integration, and support costs; the current ad-hoc system costs 
more in all these areas. 

• MC3T will also increase the effectiveness and efficiency of the 
MAGTF C4I SoS; users will be more effective at managing and 
using information. IfMC3Tfails, users will be forced to 
operate in their current ad-hoc fashion. 

6. Your Organization: 

a. How long has your organization been in operation? MCTSSA has been in 
operation since the 1970s. 

b. Does your organization follow INCOSE, IEEE, ANSI or other process 
standards? MCTSSA does not have a defined, repeatable set of process 
standards. MC3T is an attempt to introduce process standards. 

c. How long have you operated (sold, created, supported) your SoS? Since 


d. Is there anything you would like to add about your organization or SoS? 
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JTEM CTM Response to Questionnaire Version 1 

1. SoS Definition: 

a. What is your SoS? A Joint SoS test process. 

b. Where is it used? JTEM will work with Live, Virtual, and Constructive 
elements on joint distributed environment ranges. 

c. What does it do? JTEM will define a Joint SoS test process. . 

d. What are the high-level components of your SoS? DOTMLPF 

e. Are components created or managed independently, or are they managed 
centrally? JTEM will be created centrally (DOT&E), with inputs from the 
JTEM COL As the JTEM process becomes included in JCIDS and other 
acquisition processes, it will be managed and executed independently. 

f. Was your SoS built to a systems architecture model, as an automobile 
might be, or did it evolve, as the Internet has? JTEM will be built to a 
systems architecture. 

2. Requirements: 

a. Who creates requirements for your SoS? JTEM is creating SoS testing 
requirements, based on COL input. Note the JTEM charter, the 
transformation roadmap. At Rock Drills a gap was discovered in 
requirements for Joint mission test requirements. 

b. Are requirements generated within or outside your organization? JTEM 
requirements are generated internally, other than those requirements that 
come from DOD acquisition instructions (JCIDS, others). 

c. Does the requirements generator have any authority over construction, 
operation, or maintenance of the SoS? JTEM does not have any authority 
over construction, operation, or maintenance of the SoS, first, because 
JTEM is a limited-duration project, and second, because JTEM will define 
a process for use by others. 

d. Do requirements for the SoS change over time? As the JTEM COL 
matures, and JTEM events are assesses, SoS (JTEM) requirements 
change. 

e. Are requirements documented? ITEM is defining requirements as they 
are identified. 

f. Can I review your requirements? Yes, see latest documentation (Draft) 
Rock Drill Event Final Report. 

3. Validation and Verification (V&V): 

a. Is validation (“does it do the right things?”) or verification (“does it 
perform to specifications?”) required for your SoS, or for component 
systems? JTEM has conducted V&V through rock drills (tabletop 
exercises) to determine if recommended processes, at their current state, 
work. Rock drills also expose gaps and seams between current processes. 
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b. Who performs V&V for your SoS? Is this an internal or external 
organization? V&V (rock drills) was conducted internally, with 
participants from almost all services and many agencies. 

c. What are sources of V&V requirements: are they derived from system 
requirements, user requirements, or other sources? System requirements, 
doctrine, regulations, and user requirements. 

d. What are the consequences of V&V failure? “Failure” of V&V, in the 
early stages of JTEM maturation, is very unlikely, because JTEM is 
identifying requirements, testing iteratively, and exposing gaps and seams 
in models and processes. If a current version of JTEM was to fail, it 
would provide more information for analysis, and be the precursor to 
updated models and processes. 

e. How is V&V performed for your SoS? See (Draft) Rock Drill Event Final 
Report for detail. 

f. If components of the SoS change, is testing performed? Yes.. 

g. Can I view your test procedures and results reports? See (Draft) Rock 
Drill Event Final Report for detail. . 

h. Is there a process to define: 

a. test environment, 

b. test criteria, 

c. test procedures, 

d. test conduct, and 

e. results reporting? 

There will be a series of test events (USAF?) this year, and a 
follow-on next year with FCS [Future Combat System]. 

4. Support: 

a. Is your SoS supported by an internal or external organization? JTEM is 
supported internally. On execution/implementation, JTEM will be 
supported by using organizations, which may utilize internal or external 
resources. 

b. How is your SoS maintained? On execution/implementation, JTEM will 
be supported by using organizations, which may utilize internal or 
external resources. 

c. Is maintenance performed by an internal or external organization? On 
execution/implementation, JTEM will be supported by using 
organizations, which may utilize internal or external resources. 

d. Is the SoS subject to change? JTEM is evolving and expects to continue to 
evolve. 

e. How is the configuration of the SoS initially recorded, how is it checked, 
and how is it controlled? JTEM will change as the System Under Test 
(SUT) changes. User organizations will control configuration through 
current and future internal processes. 
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/ If part of the SoS is inoperative, how do users request service? On 
execution/implementation, JTEM will be supported by using 
organizations, which may utilize internal or external resources. 

g. How are users trained on functions of the SoS? On 
execution/implementation, JTEM will be supported by using 
organizations, which may [generate?] current or future processes for user 
training. 

h. Are there user, support, or other training resources I can view? There may 
be access to limited training information on the conduct of rock drills. 

i. How are support personnel trained? On execution/implementation, JTEM 
will be supported by using organizations, which may [generate?] current 
or future processes for user training. 

5. SoS Operations: 

a. What is the life cycle of your SoS? JTEM, by charter, will only last until 
[2009?]. After this point, JTEM will be supported by using organizations. 

b. Does your SoS operate continuously or intermittently? TheJTEMprocess 
will operate continuously as SoS are under test. . 

c. How long do you expect your SoS to function? JTEM is ivolving and 
expects to continue to evolve. 

d. Does your SoS have documentation? See (Draft) Rock Drill Event Final 
Report for detail. See also other JTEM documents. 

e. Can I view the documentation? See (Draft) Rock Drill Event Final Report 
for detail. See also other JTEM documents. 

f. Is there an alternative or backup to your SoS? Current disparate, 
fragmented, and ad hoc testing. 

g. What are the consequences (risk of injury or loss of life, risk of damage or 
loss of equipment, risk of financial loss, other risk) of SoS failure? Higher 
resource use (cost) for testing due to “reinvention ” of test processes for 
events; lower confidence in capability of SoS to achieve capability; 
increased possibility of Invalid assessment of SoS suitability for use 
(milestone decisions). 

6. Your Organization: 

a. How long has your organization been in operation? JTEM was chartered 
in 2006. 

b. Does your organization follow INCOSE, IEEE, ANSI or other process 

standards?_. 

c. How long have you operated (sold, created, supported) your SoS? JTEM 
was chartered in 2006 

d. Is there anything you would like to add about your organization or SoS? 
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QUESTIONNAIRE VERSION 2. 


JITC Response to Questionnaire Version 2 


1. SoS Assessment Planning 

a. Does planning for an assessment have a relationship to the scope of the 
assessment? Yes. For larger, more complex systems, or systems requiring 
multiple test venues we develop an ICEP (Interoperability Certification 
Evaluation Plan — basically, a plan for managing the testing). 
Interoperability Test Plans (ITPs) provide detailed plans/procedures for 
individual tests. Larger programs will also have a TEMP or similar 
document. For many, if not most assessments, JITC leverages off testing 
conducted by others, including use of other’s test plans/procedures. 
Usually, this involves reviewing OT [Operational Test] plans to ensure 
adequate data is collected for JITC to perform an interoperability 
evaluation. 

b. Does your organization have a planning template for assessments? Not 
sure what is meant. We have guidance/policy, there is also 
guidance/policy in CJCSI 6212.01, etc. 

c. Does your organization use plans or results of previous assessments to aid 
in planning for new events? Absolutely! 

2. SoS Assessment Resources: 

a. How are resources required to conduct the assessment (time, people, 
hardware, software, processes) identified? During planning. 

b. How are resource gaps identified and mediated? Same. 

c. How are resource conflicts identified and mediated? Same. Conflicts with 
external resources (e.g., availability of interfacing systems) can be raised 
to the MCEB[Military Communications & Electronics Board 
/Interoperability Test Panel or Joint Staff (J-6), if necessary, but this is 
rarely required. 

d. How are priorities of support established: This was covered in previous 
versions of 6212, however, it never really became an issue by itself. See 
excerpt from CJCSI 6212.01C below: 

“DISA[Defense Informatin Systems Agency] (JITC) uses the following 
organizational prioritization for testing, assessing, and certifying 
interoperability: (i) joint IT and NSS [National Security Systems] systems 
that support the unified commands, (ii) joint IT and NSS systems that are 
acquired by the Services, and (Hi) systems that are acquired by the 
Defense agencies. 

The order for functional prioritization is: (i) strategic warning and 
communication systems that support the unified commands, the Secretary 
of Defense, and the Commander-in-Chief; (ii) tactical systems that support 
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the unified commands; (iii) C2 systems that support the unified 
commands; and (iv) Combat service support systems that support the 
unified commands. ” 


(For some testing at JITC, such as standards conformance and DSN 

[Defense Switched Network], there is a simple FIFO {First In, First Out] 

queue approach. 

i. External agency priorities? 

ii. Higher Head Quaters priorities? 

iii. Other? 

3. SoS Components Evaluated 

a. How are components of the SoS identified for participation in the 
assessment? DoD 4630 and 5000 series, CJCSI 6212.01, etc. require 
systems to be interoperable, have requirements certified, and receive JITC 
Joint Interoperability Test Certification. Every system/system component 
(even legacy, unless granted a waiver) is in effect “nominated” by birth in 
the DoD community. J-6I certified capabilities documents identify 
systems/system components (e.g., in SV [Systems View]-1/2 architecture 
products). 

b. What organization(s) nominate components for assessment? The 
PM/sponsor is responsible for ensuring that interfacing systems 
participate in testing (see 6212). 

c. How are conflicts resolved? MCEB/ITP, J-6I. 

4. SoS Performance Requirements 

a. How are SoS performance requirements identified? Integrated 
architecture products and associated information (e.g., an interface may 
have an interface control document). Ideally, architecture information 
from interfacing systems would be used to validate requirements for a 
system. There are also overarching requirements such as the NCOW[Net- 
Centric Operations Warfare- Reference Model] RM (including DoD Data 
Strategy), GIGICD, GIG Architecture, etc. 

b. What organization(s) identify SoS performance requirements? 

iv. Subject Matter Experts? 

v. Test Agency? 

vi. Program Office? 

vii. End user community? 

viii. Doctrine organizations? 

ix. Joint or Service organizations? 

x. Other? All of the above, to some extent. PMs/sponsors (e.g., in the 
Army the combat developers generate initial requirements) create 
requirements that are reviewed by the JS, users, COCOMs, DISA, 
JITC, and other organizations. 

c. When are performance requirements identified: 
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xi. Before assessment request is accepted? 

xii. After approval of assessment? 

xiii. Before assessment planning stage starts? 

xiv. Other? 6212 requires that requirements be testable and 
measurable from the start, and that test planning start early in the 
life-cycle. However, sometimes requirements are not 
finalized/certified as early as desirable, and sometimes 
requirements are not as testable & measurable as they should be. 

d. How are conflicting requirements recorded and resolved? Conflicts 

should be resolved during the document development and review process. 
If conflicts are identified later, J-61 may be engaged to resolve conflicts. 

5. SoS Performance Criteria 

a. When are SoS performance criteria (i.e. speed, accuracy, timeliness, 
authenticity, repeatability...) identified? When the requirements are 
developed, although, as noted above, sometimes this information must be 
derived from supporting documents. (Integrated architecture products 
include timeliness, etc.) 

b. If performance criteria are not identified before an evaluation of the SoS is 
requested, what happens? JITC can evaluate a system based on draft 
requirements, and produce an interoperability assessment. After the 
requirements are certified the assessment letter could be upgraded to a 
certification, provided the system passes and there are no changes to the 
requirements that would require additional data not collected during 
previous testing. Requirements that are not critical, may be addressed by 
a status of “not tested, ” if other data still supports an assessment of 
critical items. (Threshold KPP requires only joint critical requirements to 
be met.) 

c. How are conflicting requirements resolved? See above. 

6. SoS Threshold and Objective values. 

a. How are threshold and objective values (i.e. 500 KPH threshold, 700 KPH 
objective; 10 meter Circular Error Probable (CEP) threshold, 5 meter CEP 
objective; less than 10 seconds threshold...) identified? NR-KPPhas 
threshold/objective values, as do KIPs and other requirements. 

b. Who participates in this identification? Same as for requirements. 

c. How are conflicting requirements resolved? Ditto. 

7. SoS Test Script 

a. What organization creates the test script? Usually, the primary test 
organization (e.g., OTA[Operational Test Agency]); JITC reviews test 
plans, procedures, scripts, etc. to ensure data collected is adequate to 
address interoperability evaluation. 
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b. What stakeholders approve the test script? Generally, the testing 
organizations and others involved in the various working groups (varies 
by type, size of system). 

c. How are conflicts resolved? By test working groups. Interoperability 
issues can always be raised to the MCEB/ITP or J-6, although that would 
be less likely for issues involving very technical details of a script. 

8. SoS Configuration: 

a. When is the SoS assessment configuration defined? JITC tests to the 
approved Information Assurance (IA) configuration. The configuration 
must represent a realistic operational environment. 

b. When is the SoS assessment configuration installed? During pre-test 
activities. 

c. Who can authorize changes to the SoS assessment configuration, and 
under what conditions? The approved IA configuration must be used. Any 
deviations would have to be reported as testing limitations, including an 
assessment of the impact they may have on interpretation of results. If 
changes are necessary to proceed with testing (e.g., there are 
showstoppers that would make continued testing a waste), the impact on 
previously collected data must be assessed and regression testing 
performed as needed to revalidate earlier results. 

9. What are milestones in the planning, conduct, and reporting of an 
assessment: 

a. Planning? 

b. Conduct? 

c. Reporting? 

d. Other? JITC uses a basic 4-step process. Requirements (identify 
requirements, ensure they are J-6 certified, testable/measurable, etc.); 
plans (ICEP, ITPs, [Interoperability Certification Evaluation Plan, 
Interoperability Test Plan] etc.); conduct (more often monitoring other’s 
testing and collecting data); reporting (detailed test reports, assessments, 
certifications, etc.). JITC, per DoDI 4630.8, also provides input to the 
OTRR [Operational Test Readiness Review]. Standards conformance or 
other assessments may also enter into the overall testing program at 
various stages. 

10. Conduct of Performance Assessment 

a. Who executes the test script: 

i. Contractors? 

ii. government civilians? 

Hi. other? A realistic environment usually requires that the typical 
user performs the operations. Typical users may be military, 
civilians, contractors, etc. Some types of systems/situations may 
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require supplementing the typical user with other types of 
personnel, as long as this does not invalidate test results. 

b. Who supervises the conduct of the assessment: 

iv. Test Agency personnel? 

v. Program Office personnel? 

vi. Combination/Other? Lead test organization, usually OTA. JITC 
“supervises ” (or monitors) testing as need to ensure 
interoperability data is adequate. 

11. Assessment Results 

a. Who records assessment results: 

i. Test Agency personnel? 

ii. Program Office personnel? 

iii. Combination/Other? Same as supervising, although “operators ” 
record some data - data collection is automated to the extent 
practicable. 

b. How are results expressed? 

i. Accurate until SoS components change? Joint interoperability 
Test Certificates state “This certification expires upon changes 
that affect interoperability, but no later than three years from the 
date of this memorandum. ” Changes that affect interoperability 
could include any or all of the following: hardware and/or 
software changes to the system, DOTMLPF changes to the system, 
or similar changes to interfacing systems. 

ii. Accurate for a period of time? Three (3) years, because most 
systems have minor updates (sometimes frequently) and, even if the 
system under test does not change, interfacing systems are likely to 
have changed, or requirements may have (should have) changed, 
etc. 

iii. Other? 

c. How are results expressed: 

i. Pass/Fail? Yes 

ii. Numerically scored? Yes 

iii. Narrative description (without score)? Yes 

iv. Other? JITC reports an overall status (e.g., threshold NR-KPP 
met); an overall status of the elements of the NR-KPP (net-centric, 
information exchange, KIP, IA, and DISR [DoD IT Standards 
Registry]compliance); status of interfaces (certified, not certified, 
not tested), and more detailed status on net-centric requirements 
(enterprise (NCES [Net-Centric enterprise Services] /COIflevel 
services/data), information/data exchanges (OV[Operational 
View]-3/SV-6), KIP and DISR [DoD IT Standards Registry] 
compliance, and IA related compliance. Depending on the 
size/complexity of requirements, some more detail may be provided 
in the certification letter (consisting of a memo and summary 
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sections), or in a detailed test report if too voluminous to include 
in the certification letter. JITC reports the degree to which 
requirements are met (e.g., all critical requirement met, all 
requirements met) and the expected operational impact (none, 
minor, moderate, major, or critical) of any discrepancies. 

12. Communications: 

a. How frequently are results reported Varies with size/complexity, length of 
testing, visibility of program, etc. Quick Look reports/briefings are 
sometimes produced, informal testing status can be produced daily, etc. 
Detailed test reports are produced after testing (sometimes by 
organizations other than JITC); certification letters are finalized and 
distributed after results have been analyzed. 

xv. Under predefined conditions? 

xvi. On demand? 

xvii. Daily/weekly? 

xviii. At the conclusion of the assessment? 

b. Is there a separate assessment report? For large/complex systems, yes. 

c. Who controls distribution of the assessment results? 

Distribution of JITC interoperability certifications is specified in 6212. 
Test reports produced by JITC are provided to the sponsor and other 
interested/authorized parties. 

13. Organization Performance Evaluation Metrics: 

a. Do you measure equipment resources used per event? 

If equipment is provided/controlled by JITC, use is monitored to some 
extent. Resources that are periodically used to support testing (part of an 
existing lab or general test capability) are not necessarily tracked by each 
individual test event (and some resources may be shared by more than one 
system during a test event). An example is NIPRNET [Non-Secure 
Internet Protocol Router Network] access. It also depends on the nature 
of the testing, overall test configuration (many involve distributed test 
resources provided by a number of organizations), etc. 

b. Personnel labor hours used per event? Yes. 

c. Duration of processes and entire event? Yes. 

d. Other resources used in planning, conducting, and reporting an event? 
Much of the interoperability testing is piggybacked on testing performed 
mostly by others, usually OT organizations. JITC tracks resources used 
for its portion of planning, conduct, and reporting. 

14. SoS Performance Metrics: 

a. Does your organization evaluate the performance of the SoS after testing? 
Whenever possible, JITC continues to observe performance by 
participation in numerous exercises, other interoperability testing that 
includes previously tested systems as participants, reports from the field 
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(from various sources), etc. When necessary, JITC reviews the previously 
determined interoperability status to determine if the status has changed, 
possibly requiring additional testing and updating any previous 
certification letters. Systems with known discrepancies, including failure 
to obtain an interoperability certification, may be reported to J-6I and/or 
the MCEB/ITP for additional action (there is a J-6I delinquency 
program/ITP watch list). JITC/J-6I also track expiring certifications; 
MCEB /ITP tracks ICTOs [Interim Certification to Operate] (part of 
qualifying for an ICTO is a plan for obtaining JITC certification, usually 
within a year). 

b. Does your organization record comments or queries from end users? After 
fielding, JITC has a 24/7 Hotline to address inquiries/issues from end 
users (and anyone else, for that matter). Appropriate action is taken, 
including deploying personnel to help resolve interoperability problems. 
(During testing, see below.) 

c. Does your organization provide feedback to program developers on end 
user comments? JITC does not normally collect end user comments after 
a system is fielded, however, appropriate organizations are notified of 
serious interoperability issues as they are identified. (See above.) User 
comments during testing are normally recorded and provided to the 
sponsor as part of the SOP for most testing. When there are discrepancies 
during testing, JITC provides an “expected operational impact” based on 
input from the user community. 
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APPENDIX C. QUALITY OF PLANNING OUTPUTS 
QUESTIONNAIRE 


This appendix consists of three sections. First, a description of the research that 
went into the methodology behind the Quality of Planning Outputs Questionnaire is 
provided. Second, the actual Quality of Planning Outputs Questionnaire is provided. 
Third, tables of the results are provided, including a table with final calculation of the raw 
scores. 

EVALUATION MEASURE: QUALITY OF PLANNING OUTPUTS 

The team considered two scales to process SME questionnaire responses. 

Osgood’s semantic differential scale [Pershing, 2000:1-7] measures subjective 
feelings indirectly by comparing extremes of the scale using words of opposite meaning. 
Respondents consider a list of paired opposite adjectives on a continuum of 5 to 11 points 
and recorded their feelings, likes, perceptions on the scale. The scale produces interval 
data that can be used in statistical analysis. Concise directions, which are understood by 
all respondents in the same way, are mandatory for effective use. The varied background 
of the respondents was valuable for their experience, but made this scale less effective for 
this questionnaire. 

A Likert [Mitchell and Jolley, 2001: 485-487] scale is useful because responses 
can be summed to provide an overall score from each respondent. Statistical analysis, 
typically a t-test, can be used to determine the validity and accuracy of the responses. 
Likert scales are often constructed of two positive responses, two negative responses, and 
a neutral response. The team constructed a scale of two positive responses and two 
negative responses, and values from 1 (worst) to 4 (best), in order to avoid any undecided 
or “fence sitter” answers. 

Determining the Score for the Evaluation Measure Quality of Planning 
Outputs 

The team created a questionnaire consisting of four questions that covered key 
areas of what constitutes the quality of planning outputs. Each question also contained a 
criterion in order to help respondents differentiate between a value of 1 to a value of 4. 

The team administered the questionnaire to one SME from MCTSSA and two 
SMEs from NAWC China Lake. Responses are recorded in Table 21, Table 22, and 
Table 23. The Table 22 response is skewed with respect to the responses in Table 21 and 
Table 23. The Table 22 respondent indicated the FCB and SCR alternatives were not as 
clearly understood, and their ability to produce quality planning products appeared to be 
reduced. 
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The raw score was calculated by summing the scores for each alternative from the 
respondents. The overall score for the evaluation measure was calculated by dividing the 
raw score by the number of questions, which were four. The equation for overall score is 

overall score = raw score / # questions. (5) 

In order to use this score with the weights utilized in AoA, the raw index was 
calculated by dividing the overall score by the number of respondents. Thus, a raw index 
will contain a value from 1 to 4. The equation for raw index is 

raw index = overall score / # SMEs. (6) 

The raw index was used in AoA to compare quality of planning outputs for each 
alternative. 

Participants 

• The SME respondent from MCTSSA was Mr. Scot Hoesly; the SME respondents 

from NAWC China Lake were Mr. Stephan Bussell and Mr. Robert Mount. 
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EVALUATION MEASURE: QUALITY OF PLANNING OUTPUTS 
QUESTIONNAIRE 


Question #1: Ability to Create Planning Documents - Does the alternative 
develop planning products (test plans, test procedures) for C4I SoS evaluations? 

Objective: Planning products are produced each time the process is executed. 
Criteria: 


Criterion 

Score 

The alternative always produces planning 
products for evaluating C4I SoS. 

4 

The alternative often produces planning 
products for evaluating C4I SoS. 

3 

The alternative sometimes produces 
planning products for evaluating C4I SoS. 

2 

The alternative never produces planning 
products for evaluating C4I SoS. 

1 


Question #2: Quality of Planning Documents - Does the alternative deliver 
planning documents that conform to standards established by relevant official 
directives and guidance? (See examples of standards below) 

Objective: The planning products conform to standards. 

Criteria: 


Criterion 

Score 

The alternative produces planning 
documents that conform to standards 
established by all Services affected by the 
SoS 

4 

The alternative produces planning 
documents that conform to the majority of 
standards in directives 

3 

The alternative produces planning 
documents that conform to standards in a 
single Service’s directives 

2 

Planning documents do not conform to 
established standards 

1 
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Question #3: Usability of Planning Documents - Does the alternative produce 
planning products that can be used for C4I SoS evaluations? 

Objective: The planning products can be used easily in an organization tasked 
with designing and executing SoS evaluation. 

Criteria: 


Criterion 

Score 

The alternative provides planning products 
that are completely usable for executing 

C4I SoS evaluations 

4 

The alternative provides planning products 
that are usable for executing C4I SoS 
evaluations 

3 

The alternative provides planning products 
that are somewhat usable for executing C4I 
SoS evaluations 

2 

The alternative provides planning products 
that are not usable for executing C4I SoS 
evaluations 

1 


Question #4: Planning Documents determination of Capability - Does the 
alternative produce outputs that measure C4I SoS capabilities? 

Objective: The planning products are effective for evaluating C4I SoS 
capabilities. 

Criteria: 


Criterion 

Score 

Planning documents ensure execution of 
the evaluation will measure all desired 
capabilities of the SoS 

4 

Planning documents ensure evaluation of 
most of the desired capabilities of the SoS 

3 

Planning documents ensure evaluation of 
some of the desired capabilities of the SoS 

2 

Planning documents ensure evaluation of 
basic data exchange and technical 
interfaces of the SoS 

1 
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QUESTIONNAIRE RESULTS. 

Table 21, Table 22, and Table 23 display the results from the questionnaires for 
each of the SMEs. Table 24 provides the raw index which was used during the AoA in 
determining the quality of planning outputs for each alternative. 


Alternative 

Question 

1 

Question 

2 

Question 

3 

Question 

4 

Score 

FEDOS 

3 

3 

4 

2 

12 

MC3T 

3 

2 

3 

3 

11 

JTEM CTM 

3 

3 

4 

3 

13 

SCR 

4 

4 

3 

3 

14 

FCB 

4 

3 

4 

4 

i 15 


Table 21. Scot Hoesly’s Responses to Quality of Planning Outputs 
Questionnaire. 


Alternative 

Question 

1 

Question 

2 

Question 

3 

Question 

4 

Score 

FEDOS 

4 

4 

4 

4 

16 

MC3T 

4 

4 

4 

4 

16 

JTEM CTM 

4 

4 

4 

4 

16 

SCR 

2 

2 

2 

2 

8 

FCB 

1 

1 

1 

1 

4 


Table 22. Stephan Bussell’s Responses to Quality of Planning Outputs 
Questionnaire. 


Alternative 

Question 

1 

Question 

2 

Question 

3 

Question 

4 

Score 


4 

3 

2 

1 



4 

3 

3 

2 

12 


4 

3 

3 

2 

12 

SCR 

4 

3 

3 

4 

14 

FCB 

4 

3 

3 

4 

14 


Table 23. Robert Mount’s Responses to Quality of Planning Outputs 
Questionnaire 
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Alternatives 

Raw 

Score 

Raw 

Index 

FEDOS 

38 

3.167 

MC3T 

39 

3.250 

JTEM CTM 

41 

3.417 

SCR 

36 

3.000 

FCB 

33 

2.750 


Table 24. Table of Raw Index Scores. 
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APPENDIX D. DESIGN ALTERNATIVES DETAILS 


This Appendix discusses alternatives generation and feasibility screening of the 

Design Alternatives phase in further detail. This Appendix begins with detailed 

descriptions of the alternatives generated during the alternatives generation phase, and is 

concluded with a detailed description of the feasibility screening process that was used to 

select feasible alternatives for the JC3M project. 

GENERA TED AL TERN A TIVES 
Exhaustive Alternative 

The “Exhaustive” approach embodied the theme of doing everything, without 
limiting the evaluation to specific capabilities, or user-directed systems or functions. 

This approach would evaluate every component of the SoS, every function, every system 
under test, every capability, and every component of the individual systems. The theory 
was that by exhaustive testing, data would be generated for every evaluation measure, 
and a complete evaluation of the SoS would be provided. 

User Defines Alternative 

The “User Defines” approach was based on the user/stakeholder determining the 
functions, capabilities, and components of the SoS under evaluation. This approach 
would evaluate only those components of the SoS, systems under test, component 
systems, and capabilities requested by the user. The theory was that by selective testing, 
the time required to plan an evaluation would decrease, the user would get a selected 
review of capabilities and functionality, and performance measures would accurately 
reflect the capabilities under review. 

“Do No Harm” Alternative 

The “Do No Harm” approach was based on the use of J1TC to determine the 
capabilities, functions and components of the SoS under evaluation. This approach would 
evaluate only those components of the SoS, systems under test, component systems, and 
capabilities identified by J1TC, in their current processes, as required to demonstrate 
interoperability. The theory was that by selective testing, investigation of and 
consultation on performance measures would be reduced, saving time in planning an 
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evaluation. The user would get a selected review of capabilities and functionality which 
would be directly related to interoperability evaluations. 

System Anomaly Report (SAR) Alternative 

The “System Anomaly Report (SAR)” approach was based on determining the 
functions, capabilities, and components of the SoS under evaluation by reference to 
anomalies or concerns received from the user community. This approach would evaluate 
only those components of the SoS, systems under test, component systems, and 
capabilities identified by the user community, and recorded in SARs, as problems in 
previous increments of the SoS. The theory was that the user community would define 
unacceptable performance of the SoS, and threshold values could thus be deduced. This 
would enable selective testing, so that investigation of and consultation on performance 
measures would be reduced, saving time in planning an evaluation. End users would get 
a selected review of capabilities and functionality focused on reported issues, and 
performance measures would accurately evaluate the capabilities, functions, and 
components of interest. 

Deliberate Method Alternative 

The “Deliberate Method” approach was based on the most effective process to 
determine the functions, capabilities, and components of the SoS under evaluation. This 
approach would exhaustively identify those components of the SoS, systems under test, 
component systems, and capabilities which are affected or are under review. The theory 
was that by using the best systems engineering approach to define the most 
discriminating testing, the evaluation would be more effective in measuring the 
performance of the SoS, systems under test, or capabilities, and the stakeholders would 
derive greater benefit from focused testing. This approach was also created in an attempt 
to define the highest performance alternative, in order to offer a markedly different 
alternative solution to compare to other alternatives. 
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Capabilities Documentation Alternative 

The “Capabilities Documentation” approach was based on the use of requirements 
documents to determine the functions, capabilities, and components of the SoS under 
evaluation. This approach would evaluate components of the SoS, systems under test, 
component systems, and capabilities that have performance measures defined by Joint 
Capabilities Integration and Development System (JCIDS) documentation, such as an 
Initial Capabilities Document, Capability Development Document, or Capability 
Production Document. The theory was that by reviewing JCIDS documentation, 
investigation of and consultation on performance measures would be reduced, saving 
time in planning an evaluation; the SoS would be evaluated in accordance with its 
documented intended purposes; and performance measures would accurately reflect the 
capabilities of the SoS. 

Program Manager Direction Alternative 

The “Program Manager Direction” approach was based on the acquisition 
Program Manager, responsible for the SoS or system, determining the functions, 
capabilities, and components of the SoS under evaluation. This approach would evaluate 
only those components of the SoS, systems under test, component systems, and 
capabilities requested by the PM. The theory was that the Program Manager would 
define threshold values, and that by selective testing, investigation of and consultation on 
performance measures would be eliminated, saving time in planning an evaluation. The 
PM, as representative of the acquisition and user community, would get a selected 
review of capabilities and functionality, and performance measures would accurately 
evaluate the issues in question. 

Test Agency Direction Alternative 

The “Test Agency Direction” approach was based on the test agency determining 

the functions, capabilities, and components of the SoS under evaluation. This approach 

would evaluate only those components of the SoS, systems under test, component 

systems, and capabilities, as determined by the test agency, to bear on the issues under 

investigation. The theory was that the test agency was an objective evaluator of both 

performance measures and SoS performance, and would provide an unbiased, accurate, 

and methodical evaluation of the SoS. Further, time could be saved in planning an 
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evaluation by reducing the need for outside consultation. This assumed the test agency 
would be more efficient in establishing test requirements than other organizations. This 
approach was also created in an attempt to offer a markedly different alternative solution 
to compare to other alternatives. 

Change Driven Alternative 

The “Change Driven” approach was based on the test agency or stakeholder 
determining what is different in the SoS configuration or employment, in order to 
determine the functions, capabilities, and components or functions of the SoS to be 
evaluated. This approach would evaluate only those components of the SoS, systems 
under test, component systems, and capabilities which have changed, in order to 
document performance variances. The theory was that by testing only changed items, 
time would be saved in planning an evaluation. The user would get a review of 
capabilities and functionality that are different as a result of and attributed to changes to 
the SoS. Further, performance measures would accurately reflect the capabilities under 
review. 

FEASIBILITY SCREENING 

After reviewing the nine alternatives from the standpoint of feasibility and 
similarity, the “User Defines” alternative was eliminated from further consideration 
because it was very similar to the “Program Manager Defines.” 

The “Do No Harm” alternative was eliminated because the team planned to 
consider the performance of baseline processes from other organizations (JTEM, MC3T, 
MCTSSA) from the start, and a primary goal of this alternative generation process was to 
consider two entirely new alternatives, rather than exclusively baseline processes. 

The Capabilities Documentation alternative was eliminated because it was similar 
to the SAR alternative in that both alternatives rely on outside documentation for 
identification of threshold values. 

The revised list of alternatives was left as Ad-Hoc, SAR, Deliberate Method, 
Program Manager Direction, and Change Driven. 
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One process of screening alternatives with multiple criteria [Blanchard and 
Fabrycky, 1998: 567] is to assign weighting factors to each evaluation measure, score 
performance on each evaluation measure, and sum scores for each alternative, which 
results in an overall scores. This method was not used because all five alternatives under 
consideration were theoretical, and there was no measurable data to compare. Simulation 
or modeling of the alternative was not employed, because the goal at this stage was not to 
compare the performance of alternatives, but to reduce the field to two alternatives. 

Another method of screening alternatives [Thuesen and Fabrycky, 2001: 567] is 
to perform paired comparisons. Paired comparisons contrast the performance of any two 
alternatives to each other on a single MOE, to determine which alternative is superior. 
Alternative A is compared to B, on a single parameter, with the result a ranking of the 
two alternatives: A > B. The process is repeated (B>C, C>D, D>E...) for each 
alternative and MOE. The property of transitivity (if A > B and B > C, then A > C) is 
applied to the results and a ranking of all alternatives can be generated. Paired 
comparison was not used to screen the alternatives because the comparison of alternatives 
is subjective, depends on expert evaluation of the alternatives, and as with weighting 
factors, there was no measurable data to compare. Simulation and modeling of the 
alternative’s performance was not employed, as stated above, because the goal was to 
reduce the field of alternatives. 

Systematic elimination [Thuesen and Fabrycky, 2001: 569] also supports the 
evaluation of the alternatives on each parameter, by eliminating alternatives that do not 
meet a minimum level of acceptability. Systematic elimination can be implemented in a 
restrictive manner by stipulating that if an alternative achieved a passing score on each 
criterion, the alternative would be retained for further development. An unrestrictive 
implementation of systematic elimination would keep any alternative that achieved a 
passing score on at least one criterion. A moderate approach would keep alternatives that 
meet more, or fail fewer, criteria. This process was also rejected because it depends on 
performance scores, which do not exist for four of the five alternatives. 

In cases where performance data are not known, a datum design comparison 

[Cross, 2000:156] can be conducted. In this process, each alternative is compared, 
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parameter by parameter, against a known quality solution. The result of each comparison 
is recorded as the same (=) performance, better than (+), or worse than (-) the baseline. 

At the completion of the comparisons the positive rankings for each alternative are 
summed, as are the negative rankings, and an overall total is calculated for each 
alternative. In this manner the qualitative comparisons of performance against the 
baseline were converted into a quantitative ranking of alternatives. The overall totals 
were used to screen out the untenable alternatives from further consideration. 

In the case of the five remaining alternatives, the datum design comparison was 
used, and the parameters for scoring were drawn from the evaluation measures that 
support the JC3M functional hierarchy. The advantage of using the datum design 
comparison is that the team understands the strengths and weaknesses of the baseline 
alternative. Thus, by comparing the proposed new alternatives with the evaluation 
measures against the baseline the team could speculate whether it will be better, worse, or 
the same as the baseline for each of these evaluation measures. Also, these evaluation 
measures were used for comparing the final five alternatives in the AoA. 

For each evaluation measure, each alternative was scored against the MCTSSA 
(FEDOS) baseline datum. The scoring criteria were as follows: (+) the alternative was 
expected to perform better than the baseline, (-) the alternative was expected to perform 
worse than the baseline, and (=) the alternative was expected to perform the same as the 
baseline. The team reviewed the known performance of the baseline against the 
projected performance of the alternative process, function by function. This ranking was 
performed initially without the aid of an SME. After evaluating each alternative, the (+) 
values were summed and the (-) values were subtracted, to calculate a final score. 
Deliberate Method and Change Driven alternatives were ranked highest, as displayed in 
Table 25. 
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Ad- 

Hoc 

SAR 

Deliberate 

Method 

PM 

Direction 

Change 

Driven 

Evaluation Measures 






Percentage of Traceable Thresholds 

= 

+ 

+ 

+ 

+ 

Percentage of Capabilities Identified 

= 

+ 

+ 

+ 

= 

Days to Plan Evaluation 

+ 

= 

= 

= 

+ 

Number of Traceable Thresholds Identified 

= 

+ 

+ 

+ 

+ 

Percentage of Shortfalls Identified 

= 

= 

+ 

= 

= 

Quality of Planning Outputs 

= 

= 

+ 

= 

= 

Number of C4I Systems Under Test 

= 

= 

= 

= 

= 

Percentage of Interfaces Identified 

= 

= 

+ 

= 

+ 

Total + 

1 

3 

6 

3 

4 

Total - 

0 

0 

0 

0 

0 

Overall Total 

1 

3 

6 

3 

4 


Table 25. JC3M System Alternatives Datum Design Comparison Matrix. 
Initial Screening of Alternatives against Baseline (FEDOS) 
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APPENDIX E. EVALUATION MEASURES DETAILS 


FUNCTIONS AND EVALUATION MEASURES 

Evaluation Measures (EM) were developed to tie to each function of the JC3M 
system, to ensure each function was not only measurable, but potentially capable of 
serving as a discriminator in considering alternative JC3M solutions. 

In the following review of the functions of the JC3M system, the EMs are 
described in terms of their attributes, using a methodology described by the U.S. Army 
Training and Doctrine Command [Borman, 1993: 30], 

Definition of the measure: A statement of the measure that includes directions for 
computation. Input data is identified for use in evaluation of alternatives. 

Dimension of the measure: How the measure is expressed in terms of scale of 
measurement and unit of measure. The scale will be one of four categories [Pariseau, 
1994], 

Nominal - Nominal data is “named” data, which can be described, but cannot be 
manipulated arithmetically. Preferences cannot be implied from the label. Examples of 
nominal data sets include red, yellow, blue; eggs, chickens, cats; Marines, civilians, 
academics. 

Ordinal - Ordinal data is “ordered” data, which can be assigned a location in a 
sequence, and a label designating that location. The interval between the values is not 
necessarily consistent. Ordinal data cannot be used to perform mathematical 
combinations or operations, other than a comparison of rank. Examples of ordinal data 
sets include First, Second, Third; A, B, C; Superior, Average, Marginal. 

Interval - Interval data has a consistent scale, but the zero point is not absolute, 
and is only stipulated for convenience. Interval data values can be added and subtracted, 
but not multiplied or divided. Example interval data sets include temperature expressed 
on the Celsius or Fahrenheit scale (which do not have an absolute zero), or dates on a 
calendar. 


165 



Ratio - Ratio data incorporates all the attributes of the other categories, as well as 
an “absolute” zero point, which indicates a complete lack of the value being measured. 
The scale is consistent, and all arithmetic operations can be performed, including the 
calculation of ratios to express a relationship between values. Examples include age, 
temperature expressed on the Kelvin scale (which does have an absolute zero), and 
physical measurements (length, weight, height.) 

Summation - Summation is the addition of a set of numbers; the result is their 
sum. Examples of “numbers” to be summed may be natural numbers, complex numbers, 
matrices. 

Limits on the range of measure: Any limits on input or output of the measure. 

Rationale for the measure: Why the measure was selected and what properties 
make it useful. 

Relevance of the measure: Circumstances in which the measure would contribute 
to the decision process. 

Associated measures: Other measures which may be used in conjunction with the 
measure, or which must be used with it to appropriately evaluate the issue. 
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(1.0) Plan C4I SoS Evaluation 

Planning is the function that drives the content of an evaluation. The planning 
function identifies the resources required to evaluate the performance of a C4I 
SoS. 

(1.1) Define Problem (Figure 48) 

Determination of the SoS capabilities and operating conditions to be considered in 
the SoS evaluation. 



Figure 48. Define Problem. 

Functional breakdown of Define Problem (1.1) with evaluation measures 

(1.1.1) Identify SoS Capabilities 

How well does this alternative define the capabilities of the SoS to be evaluated? 
EM #1: Percentage of SoS Capabilities Identified 

1. Definition of the Measure: 

This EM is calculated by dividing the count of SoS capabilities identified 
by the alternative by total SoS capabilities. Input data is the number of 
SoS capabilities. 

2. Dimension of the Measure: 
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Ratio - percentage of capabilities identified. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent 

4. Rationale for the Measure: 

This measure directly addresses the ability of the alternative to effectively 
identify capabilities. 

5. Relevance of the Measure: 

The measure may be used to compare alternative JC3M solutions under 
the same conditions. 

6. Associated Measures: 

Percentage of Operating Conditions Identified 


(1.1.1) Identify SoS Operating Conditions 

How well does this alternative identify the conditions under which the SoS will be 
evaluated? 

EM #1: Percentage of Operating Conditions Identified 

1. Definition of the Measure: 

This EM is calculated by dividing the count of SoS operating conditions 
identified by the alternative JC3M solution by total SoS operating 
conditions. Input data is the list of SoS operating conditions, including 
time of day, location, physical environment, and distance. 

2. Dimension of the Measure: 

Ratio - percentage of operating conditions identified. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent. 

4. Rationale for the Measure: 
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This measure directly addresses the ability of the alternative to effectively 
identify operating conditions, and can be considered an indication of the 
flexibility of the alternative to process complex SoS. Understanding the 
environment in which the SoS will be used will support testing to the same 
requirements. Does this alternative identify the fleet or field environment 
the SoS was intended to perform in? 

5. Relevance of the Measure: 

The measure may be used to compare alternative JC3M solutions under 
the same conditions. 

6. Associated Measures: 

Percentage of SoS Capabilities Identified 
(1.2) Define Components (Figure 49) 

Identify the System(s) Under Test (SUT) - those individual systems that comprise 
the SoS, and will be included in the 
evaluation. 



Figure 49. Define Components. 

Functional breakdown of Define Components (1.2) with evaluation measures 
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(1.2.1) Identify Systems Under Test 

How well does this alternative identify SUT required for the SoS evaluation? 

EM #1: Percentage of SUT Identified 

1. Definition of the Measure: 

This EM is calculated by dividing the count of SUT identified by the 
alternative by total SUT count. Input data is the number of SUT, resident 
in the SoS, required for the evaluation. 

2. Dimension of the Measure: 

Ratio - percentage of SUT identified. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to identify SUT. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

None. 

(1.2.2) Identify SUT Capabilities 

How well does this alternative identify capabilities the SUT(s) must perform to 
support the SoS evaluation? Function 1.1.1 identifies the SoS Capabilities; this function 
identifies the capabilities of the system(s) participating in the evaluation. 

EM #1: Percentage of SUT Capabilities Identified 

1. Definition of the Measure: 
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This EM is calculated by dividing the count of SUT capabilities identified 
by the alternative by total SUT capabilities. Input data is the number of 
SUT capabilities. 

2. Dimension of the Measure: 

Ratio - percentage of capabilities identified. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to identify 
capabilities. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

Percentage of SoS capabilities identified. 

(1.2.3) Identify SUT Interfaces 

How well does this alternative identify interfaces between SUT which are 
required to support the SoS evaluation? 

EM #1: Percentage of SUT Interfaces Identified 

1. Definition of the Measure: 

This EM is calculated by dividing SUT interfaces identified by the 
alternative by the total SUT interfaces. Input data is the number of SUT 
interfaces. 

2. Dimension of the Measure: 

Ratio - percentage of interfaces identified. 

3. Limits on the range of Measure: 
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The output can assume any value from 0 to 100 percent. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to identify interfaces. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions 

6. Associated Measures: 

None 

(1.3) Define Evaluation Criteria (Figure 50) 

Determination of the Resources required to conduct the evaluation, and criteria 
used when evaluating the SoS. 



Figure 50. Define Evaluation Criteria. 

Functional breakdown of Define Evaluation Criteria (1.3) with evaluation measures 

(1.3.1) Identify Required Resources 

How well does this alternative identify resources (people, time, equipment, etc.) 
required to conduct the SoS evaluation? 

EM #1: Percentage of Required Resources Identified 
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1. Definition of the Measure: 

This EM is calculated by dividing the count of resources identified by the 
alternative JC3M solution by total resources required. Input data is the 
number of resources required for SoS evaluation, including personnel, 
facilities, C4I systems, training, and services. 

2. Dimension of the Measure: 

Ratio - percentage of resources identified. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to identify required 
resources. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

None 

(1.3.2) Define Measures 

How well does this alternative identify measures of performance when evaluating 
the SoS? 

EM #1: Percentage of Traceable Measures 
1. Definition of the Measure: 

This EM is calculated by dividing the number of measures (traceable to 
stakeholder requirements) generated by the alternative, by the number of 
measures generated by the alternative. Input data is the list of stakeholder 
requirements. 
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2. Dimension of the Measure: 


3. 

4. 

5. 

6 . 

EM# 

1 . 


2 . 

3. 

4. 

5. 


Ratio - percentage of traceable measures identified. 

Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent.. 

Rationale for the Measure: 

This measure addresses the ability of the alternative to identify measures. 
Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

Associated Measures: 

None. 

2: Number of Traceable Measures 
Definition of the Measure: 

Number of Traceable Measures is calculated by summing the number of 
measures, traceable to stakeholder requirements, generated by the 
alternative. Input data is the list of stakeholder requirements. 

Dimension of the Measure: 

Interval - total of traceable measures identified. 

Limits on the range of Measure: 

Output is a real number greater than or equal to zero. 

Rationale for the Measure: 

The measure addresses the ability of the alternative to perform a core 
function of JC3M, and identify measures. 

Relevance of the Measure: 


174 



The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

None. 

(1.3.3) Create Test Plan 

How well does the alternative create test plans? 

EM #1: Quality of Test Plan 

1. Definition of the Measure: 

This EM is calculated by assigning an overall quality level to the test plan 
produced by the alternative. Input data is the list of JC3M internal and 
external documents. 

2. Dimension of the Measure: 

Ordinal - Low, Medium, High. 

3. Limits on the range of Measure: 

Output is a qualitative ranking of the outputs. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to create effective test 
plans. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions 

6. Associated Measures: 

None 

(1.4) Ensure Evaluation Readiness (Figure 51) 

Review plans and assumptions to ensure required resources are available, or that a 
remediation plan is in place. Identification of shortfalls will determine if the evaluation 
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can continue, or if additional planning is required. The review is designed to ensure 
expectations and risks are correctly identified. Before the evaluation begins, all 
stakeholders must agree that requirements, expectations, and risks are accurately 
identified, conflicts resolved or noted, and remediation plans are in place. When the 
review is completed, measures are gathered on the conduct of the entire planning phase. 



Figure 51. Ensure Evaluation Readiness. 

Functional breakdown of Ensure Evaluation Readiness (1.4) with evaluation measures 

(1.4.1) Identify Shortfalls 

How well does this alternative identify gaps between resources required to 
evaluate the SoS and the resources included in the plan for SoS evaluation? 

EM #1: Percentage of Shortfalls Identified 

1. Definition of the Measure: 

This EM is calculated by dividing the number of resource shortfalls 
identified by the alternative, by the total number of shortfalls. Input data 
to be used is the list of required resources and known shortfalls. 

2. Dimension of the Measure: 

Ratio - percentage of resource shortfalls identified. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent. 
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4. Rationale for the Measure: 


This measure addresses the ability of the alternative to identify missing or 
incomplete resources. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

None. 


(1.4.2) Conduct Review 

How long does the alternative take to review the evaluation plan with the 
stakeholders? 

EM #1: Review Duration 

1. Definition of the Measure: 

This EM is calculated by summing the total duration of the evaluation plan 
review. 

2. Dimension of the Measure: 

Ratio - time in hours. 

3. Limits on the range of Measure: 

Output is a real number greater than or equal to zero. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to conduct readiness 
reviews. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 
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6. Associated Measures: 


Percentage of Documents Review Quality of JC3M Documents 
EM #2: Percentage of documents reviewed 

1. Definition of the Measure: 

This EM is calculated by dividing the number of JC3M internal and 
external documents (to included SE Artifacts) reviewed by the total 
number of JC3M internal and external documents. 

2. Dimension of the Measure: 

Ratio - percentage of documents reviewed. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to review JC3M 
documents effectively. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

Review Duration 

Quality of JC3M Documents 
(1.4.3) Planning Results 

How long did this alternative take to execute the planning phase take, and what 
were the results? 

EM # 1: Hours to Plan Evaluation 
1. Definition of the Measure: 
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At the completion of the review, this EM is calculated by summing the 
labor man-hours required to plan an evaluation. Input data is the duration 
of each Planning task, as performed by the alternative. Note that this data 
was not used to directly compare the performance of alternatives. Rather, 
this data was used in the LCCE to calculate the personnel costs associated 
with performance of tasks by an alternative. 

2. Dimension of the Measure: 

Summation - duration of evaluation planning. 

3. Limits on the range of Measure: 

Output is a real number greater than or equal to zero. 

4. Rationale for the Measure: 

This measure addresses the labor hours used in the planning phase of 
JC3M system, and indirectly addresses the duration of JC3M system 
evaluations. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

Days to Plan Evaluation. 

Quality of Planning Outputs. 

EM # 2: Days to Plan Evaluation 

1. Definition of the Measure: 

At the completion of the review, this EM is calculated by summing the 
days required to plan an evaluation. Input data is the duration of each 
Planning task, as performed by the alternative. 

2. Dimension of the Measure: 
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Ratio - duration of evaluation planning in days. 

3. Limits on the range of Measure: 

Output is a real number greater than or equal to zero. 

4. Rationale for the Measure: 

This measure addresses the duration of the planning phase of JC3M 
system, and indirectly addresses the duration of JC3M system evaluations. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

Hours to Plan Evaluation. 

Quality of Planning Outputs. 

EM #3: Quality of Planning Outputs 

1. Definition of the Measure: 

At the completion of the review, this EM is calculated by assigning an 
overall quality level to the planning documents produced by the 
alternative. Input data is the list of JC3M system internal and external 
documents. 

2. Dimension of the Measure: 

Ratio - from 1 - 4 (Likert Scale.) 

3. Limits on the range of Measure: 

Output is a qualitative ranking of the JC3M internal and external 
documents. 

4. Rationale for the Measure: 
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This measure addresses the ability of the alternative to create effective 
JC3M documents. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

Hours to Plan Evaluation 

Days to Plan Evaluation 
(2.0) Evaluate Capability of C4I SoS 

Test agencies will use JC3M to evaluate the performance of the SoS. JC3M does 
not include new processes, and instead relies on existing Joint and Service processes for 
the conduct of C4I SoS evaluations. Because JC3M relies on existing processes, this 
function is not explicitly modeled, nor are there defined evaluation measures. 

Modeling and simulation is highly recommended as part of the conduct of any 
C4I SoS evaluation. 


181 



(3.0) Report Results 

Test and acquisition agencies will use JC3M to report the results of an evaluation 
because the existing process which do in fact clearly indicate pass or fail based on 
meeting or not meeting a set of thresholds can no longer be used. Therefore, the reporting 
system needs to be different to reflect the results in a form of “here is what the system 
does” and allow the end-user to say if it’s good enough. 

The evaluation and the planning for the evaluation do not focus on creating the 
threshold values but rather focus on finding and defining more reasonable evaluation 
measures that make more sense to an end-user, but not defining “how good is good.” 

(4.0) Adaptability (Figure 52) 



Figure 52. Adaptability. 

Non-functional breakdown of Adaptability (4.0) with Evaluation Measures 


(4.1) Input System Flexibility 

How well does the alternative support varying types of C4I SoS? 

EM #: 1 Types of C4I SoS 

1. Definition of the Measure: 

This EM is calculated by recording the type of SoS the alternative can 
evaluate. Input data is the list of SoS types. 

2. Dimension of the Measure: 

Nominal - Joint, Service, both. 

3. Limits on the range of Measure: 
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Output is a list of SoS types supported by the alternative. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to evaluate varying 
types of C4I SoS. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

Number ofC4I SUT 
(4.2) Input System Elasticity 

How does the alternative respond when the SoS evaluation changes scope? 

EM #: 1 Elasticity of Labor 

1. Definition of the Measure: 

This EM is calculated dividing the percent change in systems under test by 
the percent change in labor hours to conduct the planning phase of JC3M. 

2. Dimension of the Measure: 

Ratio - percentage change. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to plan C4I SoS of 
varying sizes. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 
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6. Associated Measures: 


Elasticity of Duration 
EM # 2: Elasticity of Duration 

1. Definition of the Measure: 

This EM is calculated dividing the percent change in systems under test by 
the percent change in duration to conduct the planning phase of JC3M. 

2. Dimension of the Measure: 

Ratio - percentage change. 

3. Limits on the range of Measure: 

The output can assume any value from 0 to 100 percent 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to plan C4I SoS of 
varying sizes. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

Elasticity of Labor 
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(5.0) Usability (Figure 53) 



Figure 53. Usability. 

Non-functional breakdown of Usability (5.0) with evaluation measure. 

(5.1) Implementation of JC3M 

What resources are required to implement the alternative? What is the duration of 
the implementation process, i.e. the time required to replace the baseline process with the 
alternative? 

EM # 1: Time to implement JC3M 

1. Definition of the Measure: 

This EM is calculated by summing the time required to implement the 
alternative. Input data is based on calculated implementation time. 

2. Dimension of the Measure: 

Ratio - duration of implementation. 

3. Limits on the range of Measure: 

Output is a real number greater than or equal to zero. 

4. Rationale for the Measure: 
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This measure addresses the duration of implementation, and indirectly 
addresses the cost of implementing the alternative. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

None. 

(6.0) Repeatability (Figure 54) 

One goal of JC3M is to produce the same outputs for each SoS evaluation. If 
JC3M consistently produces products at the same level of quality, for a variety of 
scenarios, it has high repeatability. As scenarios vary, what is the effect on the quality of 
the output from the alternative? 

Repeatability 

6.0 


EM: Variance in Quality 


Figure 54. Repeatability. 

Non-functional breakdown of Repeatability (6.0) with evaluation measure 
EM # 1: Variance in Quality 

1. Definition of the Measure: 

This EM is calculated by assigning an overall quality level to the products 
produced by the alternative in varying scenarios. This EM is not just 
measuring the quality level, but the difference in the quality level between 
the different scenarios. Input data is the list of JC3M internal and external 
documents. 

2. Dimension of the Measure: 
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Ordinal - Low, Medium, High. 

3. Limits on the range of Measure: 

Output is a qualitative ranking of the outputs. 

4. Rationale for the Measure: 

This measure addresses the ability of the alternative to create effective 
documents. 

5. Relevance of the Measure: 

The measure may be used to compare alternatives under the same 
conditions. 

6. Associated Measures: 

None. 
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APPENDIX F. CORE IDEFO DIAGRAMS 


FEDOS IDEFO Diagram 


SUT Versions Numbers 
Test Dates 
5UT List 
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List of Proposed Tests 
Network Arch Diagrams 
List of Proposed Eval Metrics 


Test Schedule 
System Delivery Dates ! 

Staffing Plan 1 


Elicit Stakeholder 
Requirements for 
C4ISP Systems 
Configuration B... 




Confirmed Test Dates 


Confirmed List of Requirements 


sutVioI 


hfirmed List 


F.2 


Generate an 
Evaluation Plan 


Evaluation Plan 


. r 

] Team Lead 
Log Manager 


\ 

Config Mgr/Qual Cntrl 


F.3 


Develop a Test 
Plan and Test 
Procedures 


-► Test Plan 


-► Test Procedures 



\ Planning Lead 

N 

Config Mgr/Qual Cntrl 
DCACT Lead "A" 

DCAT Lead "B" 

Tech Lead 

test Coord Site Lead 
Sr. Tech Writer 


Figure 55. FEDOS IDEFO Diagram. 

This diagram illustrates only the FEDOS tasks performed in planning a C4I SoS Evaluation. Parentheses (“tunnels”) on the left indicate 
stakeholder inputs or previous tasks. Test Plans and Procedures connect to the complete FEDOS C4I SoS evaluation process. 
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MC3T IDEFO Diagram 


MC3M Capabilities Package \ -► 


Consolidated Requirements Review (CRA) 


In Band and Out-of-Band Topologies 
Test Cases/In Band and Out-of-Band Topologies 


Technical Proposal 



Data Analysis Plan 
Data Capture Plan 

__Draft In Band Topology 
Draft Out of Band Topology 
Test Cases 

Topology Review Results 
MC3M Technical Proposal 


Event Plan 


Data Collection and Analysis Lead "A" (DDACT) 

Logistics Manager 


Configuration Manager/Quality Control 
Sr. Technical Writer 
technical Lead 
test Coordinator Site Lead 
Lead C4I Operator 
Planning Lead 
Sr. Program Manager 


Figure 56. MC3T IDEFO Diagram. 

This diagram illustrates only the MC3T tasks performed in planning a C4I SoS Evaluation, and reflects MC3T naming and numbering 
conventions. Tunnels on the left indicate tasks and inputs to planning; tunnels on the right are used to indicate personnel inputs. 
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JTEM CTM IDEFO Diagram 


Approved AoA Plan 


Capability Dev, Document (CDD) 
Initial Capabilities Document (ICD) 
Joint Capabilities Document (JCD) 
Analytical Baseline (Scenario, CONOPS, Data) 
JOpsC Family (JOC, JFC, JIC) 
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temp' 

Test & Evaluation Strategy 
UJTL 

Capability Prod, Document (CPD) 
Joint Capability Areas 



Joint DOTMLPF Change Recommendation — 
Enterprise JME LVC Foundation Model (JFM) — 


Program Introduction (PID) 


Statement of Capability (SOC) 


Test Plan 



-> Joint Capability Evaluation Report (JCER) 



tTM,3 




Implement LVC 
Distributed 




C7 

■“Test Plan 


Environment 




Figure 57. JTEM-CTM IDEFO Diagram. 

This diagram illustrates only the JTEM CTM tasks performed in planning a C4I SoS Evaluation, utilizes JTEM CTM naming and numbering 
conventions, and may change as the JTEM CTM matures. 
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FCB IDEFO Diagrams 



Figure 58. FCB IDEFO Diagram. 

This diagram illustrates only the FCB tasks performed in planning a C4I SoS Evaluation. Tunnels on the left indicate inputs to planning; Final 
Test Plan on the right was proposed for use with existing C4I SoS evaluation processes. “FCB Measurements” is the input from the C2 FCB. 
This is in comparison to the “SCR Measures” input seen in Figure 60. 
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FCB Evaluation Measurements 



Tech Lead 


Figure 59. FCB’s Analyze Measures task 

This figure illustrates only FCB task 1.3.2.1 and how measures proposed by the FCB, and other inputs, would be analyzed by the test 
organization, and converted into metrics for use in the planning of a C4I SoS evaluation. 1.3.2.1 is a subset of FCB task 1.3 “Define Criteria” 
as seen in Figure 59. 
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SCR IDEFO Diagrams 



Figure 60. SCR IDEFO Diagram. 

This diagram shows illustrates only the SCR tasks performed in planning a C4I SoS Evaluation. Tunnels on the left indicate inputs to 
planning; Final Test Plan on the right was proposed for use with existing C4I SoS evaluation processes. “SCR Measurements” is the input to 
this task, generated within the test organization. This is in comparison to the “FCB Measures” input seen in Figure 58. 
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Desired Date for Test Results delivery 
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Thread Architecture Diagrams - 
SoS Capabilities £ 


System Level Documents ^ 


SCR Measurements - 


Comments on Metrics and Grading Process - 
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Figure 61. SCR’s Define Measures task 

This figure is a subset of SCR Task 1.3 in Figure 61, and illustrates a detailed view ofhow the SCR alternative would Define Measures. Inputs 
would be utilized by the test organization, and converted into metrics for use in the planning of a C4I SoS evaluation. 
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APPENDIX G. EVALUATION MEASURE: PERCENTAGE OF 

TRACEABLE MEASURES 


Percentage of Traceable Measures (PTM) was the EM for the Define Measures 
(1.3.2) function of the JC3M value hierarchy. The objective of the Define Measures 
function was to determine how well an alternative identifies measures of performance 
(MOP) when evaluating the SoS. An EM should not be confused with a MOP. EMs 
were measures created by this team to gauge functions of the JC3M system. MOPs are 
measures used to gauge performance of a C4I SoS. 

Definition of PTM: 

This EM was calculated by dividing the number of measures (traceable to 
stakeholder requirements) generated by the alternative, by the number of measures 
generated by the alternative. 


PTM = 


# Traceable Measures Created 
# Total Measures Created 


However, the team came to the conclusion that it was not feasible to calculate the 
PTM EM as it was defined because that would entail exercising each of the alternative 
systems and developing MOPs for each alternative. Therefore, a proxy measure was 
developed that could serve as an indicator to the performance of Define Measures 
function. 

Proxy Definition of PTM: 

This EM was calculated by taking the number of authoritative sources that were 
considered in the process and then dividing by the total number of authoritative sources 


available for the SoS. 


PTM = 


# Authoritative Sources Considered 

# Authoritatative Sources Available 


The concept was that analysts performing the Define Measures function should 
consider all available and appropriate documentation for the SoS. Considering all 
available documentation helped ensure that all requirements and testable capabilities 
were captured in the process and subsequently MOPs were defined for each. The team 
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assumed that considering a wider set of authoritative sources would yield a higher 
number of requirements and capabilities, and in turn provide more metrics that are 
traceable. The team recognized that a Program Manager (PM) might be resistant to other 
sources used to define requirements for an SoS component system, when a CDD or ORD 
may have been used used for system-level requirements. 

Figure 63 illustrates how measures are traceable to requirements for systems in 
acquisition programs. However, the JC3M problem statement is to consider evaluating 
SoSs that have been constructed or assembled with existing fielded systems. While 
component systems in the SoS do have CDDs or ORDs, the assembled SoS does not have 
CDDs, ORDs, or an overarching integrated architecture. This fact drives the need to 
obtain overall SoS requirements and measures from all relevant sources. 


198 




Figure 62. Performance Measure Traceability. 

Performance Measures Trace Back to System Capabilities and Requirements [From 
Meyer, 2007], 

Figure 63 illustrates how performance measures can be derived using an 
expanded set of authoritative sources. The next section describes the list of documents 
derived using various DoD Directives and Instructions. 
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Figure 63. Fielded IT & NSS Interoperability Process. 

Capability-focused, effects-based interoperability process for addressing operational 
warfighting interoperability and supportability issues for fielded IT and NSS [From DoDI 
4630.8,2004:26], 
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Building the PTM Denominator 

The proxy definition for PTM states the total number of authoritative sources 
available for the SoS is the denominator of the measure. The team assembled a list of 
relevant documents from three categories: 1) reference documents, 2) system level 
documents and 3) overarching DoD documents. The reference documents include DoD 
Instructions (DoDI), DoD Directives (DoDD), and Combined Joint Chiefs of Staff 
Instructions (CJCSI.) The reference documents apply to all IT and NSS and provide 
specific requirements in determining interoperability needs. The system level documents 
are specific to any given system under test, and include system specific capabilities 
documents. Overarching DoD documents include Joint documents that are applicable 
across Services and or across different C4I systems; these include Joint Concept 
documents, Joint Doctrine, and integrated architectures, 
a. Reference Documents 


1. DoDD 5000.1, DoDI 5000.2 

2. DoDD 4630.5 and DoDI 4630.8 

3. DoDI 5200.40 

4. DoDD 8500.01E 

5. CJCSI 3170.01F 

6. CJCSI 6212.01D 

7. NCOWRM 


b. Design Artifacts 

1. JCIDS Capabilities Documents 

i) Joint Capabilities Document (JCD) 

ii) Initial Capabilities Document (ICD) 

iii) Capabilities Development Document (CDD) and Capabilities Production 
Document (CPD) 
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iv) Capstone Requirements Document (CRD) 

2. System Threat Assessment (STA) 

3. Analysis of Alternatives (AO A) 

4. Information Support Plan (ISP) 

5. Test and Evaluation Master Plan (TEMP) 

6. Test Plans 

7. Data Management and Analysis Plans (DMAP) 

8. Reports from T&E, exercises, and readiness reports 

9. Combatant Commanders Input and Issues 

c. Overarching DoD Documents 

1. Joint Doctrine 

2. Universal Joint Task List (UJTL), Service task lists, and Mission Essential Task 
Lists (METLs) 

3. Joint Operating Concepts (JOC) 

4. Joint Functional Concepts (JFC) 

Description of Applicability of Documents 

The list was built using guidance from [DoDD 4630.5, 2004: 3] which states that 
all IT and NSS interoperability and supportability needs shall be derived using Joint 
Operating Concepts, Joint Functional Concepts, and associated integrated architectures 
and shall be updated as necessary throughout the system's life. IT and NSS 
interoperability and supportability needs, for a given capability, shall be identified 
through the following: 

1. The Defense Acquisition System (as defined in the 5000 series of DoD issuances) 

2. The Joint Capabilities Integration and Development System (JCIDS) process. 
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3. The Doctrine, Organization, Training, Materiel, Leadership and education, 

Personnel and Facilities (DOTMLPF) change recommendation process. 

In addition, Universal Reference Resources (URRs) are resources that must be 
consulted when building integrated architecture products for IT and NSS. URRs are: 

. .DoD Architecture Framework; DoD Core Architecture Data Model; Universal Joint 
Task List; Technical Reference Model; Global Information Grid (GIG) Architecture; 

DoD Net-Centric Data Strategy; DoD Metadata Registry; NCOW RM; and the DISR” 
[DoDD 4630.5, 2004: 57], 

“Integrated Architectures shall be used as the basis for assessment and analysis to 
characterize interoperability needs for a given capability. Solutions to the identified needs 
may be materiel or non-materiel solution sets, or both. Interoperability needs shall be 
documented in applicable capabilities documents and the NR-KPP. System information 
and interoperability dependencies and supportability requirements shall be documented in 
an ISP. Regardless of the solution (materiel or non-materiel), interoperability and 
supportability shall be tested and verified prior to operational use or fielding” 

[DoDI 4630.8, 2004: 22], 

The remainder of this section provides more detail on the list of documents. 

DoDD 5000.1 

This directive applies to all DoD acquisition programs. DoD policy is to translate 
operational needs into stable, affordable programs (e.g., through integrated product & 
process development, long-range investment plans, risk management, use of a “cost as an 
independent variable” approach, performance specifications), acquire quality products 
(e.g., based on competition, test & evaluation, modeling & simulation, past performance, 
experience in domain, mature software process), organize for efficiency & effectiveness 
(e.g., streamlined organization, trained acquisition corps, teams, tailoring, automated 
acquisition information corps, teams, tailoring, automated acquisition information) and 
make use of test and evaluation criteria as the basis of managing an acquisition program 
[DoDD 5000.1,2003:2], 
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Enclosure 1 to the directive provides additional policies on independent 
operational test agencies, information assurance, information superiority, integrated test 
and evaluation, and interoperability [DoDD 5000.1, 2003: 7], 

DoDI 5000.2 

This instruction implements DoDD 5000.1, and applies primarily to defense 
technology projects and acquisition programs, but some requirements are solely for 
Major Defense Acquisition Programs and Major Automated Information Systems. The 
instruction “establishes a simplified and flexible management framework for translating 
mission needs and technology opportunities, based on approved mission needs and 
requirements, into stable, affordable, and well-managed acquisition programs” [DoDI 
5000.2,2003: 1], 

The instruction identifies test and evaluation procedures for major acquisition 
programs and major automated information systems and provides procedures for 
assessing information interoperability, C4I support, and information assurance. These 
procedure are limited to ACAT I or IA programs and special interest programs 
designated by the Assistant Secretary of Defense for Command, Control, 
Communications, and Intelligence (ASD C3I.) The instruction also establishes 
procedures for integrated architectures and defines the responsibilities of a PM on 
development, operational, interoperability, and information assurance tests. 

DoDD 4630.5 

This directive defines a capability-focused, effects-based approach for IT and 
NSS interoperability and supportability across DoD. The directive establishes Net-Ready 
Key Performance Parameters (NR-KPP) to assess attributes required for the technical 
information exchange and end-to-end operational effectiveness of the exchange [DoDD 
4630.5,2004: 1], 

DoDI 4630.8 

This instruction implements a capability-focused, effects-based approach to IT 
and NSS interoperability and supportability throughout DoD in order to ensure their life- 
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cycle interoperability and supportability, and implements NR-KPP [DoDI 4630.8, 2004: 
25], 


DoDI 5200.40 

This instruction describes policies, processes, and the importance of IT security 
certification and accreditation in the acquisition of IT systems [DoDI 5200.40, 1997: 1], 

DoDD 8500.1E 

This directive establishes policy for Information Assurance (IA) and assigns 
responsibilities to achieve DoD IA and supports the evolution to network centric warfare 
[DoDD 8500.IE, 2002: 1], 

CJCSI 3170.01F 

This instruction describes JCIDS [CJCSI 3170.01F, 2007] and the identification, 
assessment, and prioritization of joint military capabilities needs. The instruction 
implements a capabilities based approach for identifying gaps in existing capabilities and 
developing new capabilities. The instruction also provides guidelines for JCD, ICD, and 
CDD production and use, as well as CPD and DOTMLPF change recommendations. 

CJCSI 6212.01D 

This instruction identifies information needs, dependencies, and interfaces for IT 
and NSS programs in all acquisitions, focusing on net-readiness, interoperability, 
information supportability, and information sufficiency concerns. The instruction 
provides guidance on the use of the Information Support Plan (ISP) and interoperability 
evaluation and certification procedures [CJCSI 6212.01D, 2006]. 

1. Appendix A to Enclosure C provides guidance on the use of the ISP (C4ISP) 
Assessment tool and detailed ISP review procedures [CJCSI 6212.01D, 2006: C-A-l], 

2. Appendix B to Enclosure C Provides guidelines for a Tailored Information 
Support Plan (TISP), including waivers, content, and certification [CJCSI 6212.01D, 
2006: C-B-l], 

3. Enclosure D defines the submission of NR-KPP documentation and describes 
JCIDS and ISP Document considerations; JCD, ICD, CDD, CPD, and ISP production; 
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the interoperability certification process; Global Information Grid (GIG) Key Interface 
Profile (KIPs); and IA requirements [CJCSI 6212.01D, 2006: D-l], 

4. Enclosure E provides guidelines for Joint interoperability testing and 
certification [CJCSI 6212.01D, 2006: E-l], 

Joint Doctrine 

There are 76 Joint Doctrine Publications covering Personnel, Intelligence, 
Operations, Logistics, Plans, and Communications Systems support. Applicable Joint 
Doctrine Publications will be reviewed depending on the system under test. 

Universal Joint Task List (UJTL), Service task lists, and Mission Essential 
Task Lists (METLs) 

The Universal Joint Task List (UJTL) [CJCSM 3500.04D, 2005], when 
augmented with Service task lists, is a comprehensive list of tasks, conditions, measures, 
and criteria supporting all levels of the Department of Defense in executing the National 
Military Strategy. Mission Essential Task Lists (METL) organize tasks by mission. 

Thus a METL is a compilation of tasks, with associated standards and conditions. The 
UJTL, Service Task lists METL will be used to identify tasks that the system under test, 
as part of the SoS, must perform. 

Applicable Joint Operating Concepts (JOC) and Joint Functional Concepts 

(JFC) 

[DoDD 4630.5, 2004: 13] states all IT and NSS interoperability and 
supportability needs shall be derived using Joint Operating Concepts and Joint Functional 
Concepts. JOCs and JFCs applicable to the C4I SoS under evaluation should also be 
reviewed. There are currently 6 JOC publications. 

1. Homeland Defense and Civil Support [DoD JOC HDCS, 2006] 

2. Deterrence Operations [DoD JOC DO, 2006] 

3. Major Combat Operations [DoD JOC MCO, 2006] 

4. Stability Operations [DoD JOC SO, 2006] 

5. Irregular Warfare [DoD JOC IW, 2007] 

6. Military Support to Shaping Operations [DoD JOC MSSO, 2007] 

There are currently 8 JFC publications: 
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1. Training [DoD JFC T, 2007] 

2. Force Management [DoD JFC FM, 2005] 

3. Net Centric [DoD JFC NC, 2005] 

4. Battlespace Awareness [DoD JFC BA, 2003] 

5. Command and Control [DoD JFC C2, 2007] 

6. Force Application [DoD JFC FA, 2004] 

7. Focused Logistics [DoD JFC FL, 2003] 

8. Protection [DoD JFC P, 2004] 

Determining the Score for the PTM Evaluation Measure 

If an alternative considers every document in the comprehensive list of 
authoritative documents discussed in the previous section it receives a score of 100%. 
Table A-l contains the comprehensive list of authoritative documents, weights for each 
document, and the score for each alternative. If an alternative uses a document in its 
process then the alternative receives the complete score for that document. 

Based on a number of 25 authoritative sources, if each source were weighted 
equally each would have a score of 4. However, the team felt that there were some 
sources that would contribute more to the Define Measures function than others. 
Therefore, scores were derived on a scale of 2, 4, and 8. A score of 2 was given to 
sources that were considered by the team to have the least contribution to deriving 
measurable requirements yet still essential. Scores of 4 were given to sources that were 
cited in references as being essential. For example, DoDD 4630.5 [2004: 13] states that it 
is DoD policy that interoperability and supportability needs shall be derived using Joint 
Operating Concepts and Joint Functional Concepts. Scores of 8 were given to all the 
JCIDs Capability documents which were deemed as the most important sources by the 
team. 
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Authoritative Source 

Score 

Baseline 

CTM 

MC3T 

FCB 

SCR 

a. Reference Documents (Weight incl. 1-7) 

16% 

n/a 

16% 

0 % 

16% 

16% 

l.DoDD 5000.1 and 5000.2 

2% 

n/a 

2% 

0 % 

2% 

2% 

2. DoDD 4630.5 and DoDI 4630.8 

2% 

n/a 

2% 

0 % 

2% 

2% 

3.DoDI 5200.40 

2% 

n/a 

2% 

0 % 

2% 

2% 

4. DoDD 8500.1 (IA) 

2% 

n/a 

2% 

0 % 

2% 

2% 

5. CJCSI 3170 (J8) 

2% 

n/a 

2% 

0 % 

2% 

2% 

6. CJCSI 6212.01D (J6) 

2% 

n/a 

2% 

0 % 

2% 

2% 

7. NCOW Reference Model 

4% 

n/a 

4% 

0 % 

4% 

4% 

b. Design Artifacts (Weight incl. 1-19) 

68% 

n/a 

60% 

56% 

56% 

60% 

1. JCIDS Documents (Weight incl. i-iv) 

32% 

n/a 

32% 

32% 

32% 

32% 

i) JCD 

8% 

n/a 

8% 

8% 

8% 

8% 

ii) ICD 

8% 

n/a 

8% 

8% 

8% 

8% 

iii) CDD and or CPD 

8% 

n/a 

8% 

8% 

8% 

8% 

iv) CRD 

8% 

n/a 

8% 

8% 

8% 

8% 

2. STA 

4% 

n/a 

4% 

0% 

4% 

0% 

3. AoA 

4% 

n/a 

0% 

0% 

4% 

0% 

4. ISP 

4% 

n/a 

0% 

0% 

4% 

4% 

5. TEMP 

8% 

n/a 

8% 

8% 

8% 

8% 

6. Test Plans 

4% 

n/a 

4% 

4% 

0% 

4% 

7. Data Management and Analysis Plan 
(DMAP) 

4% 

n/a 

4% 

4% 

0% 

4% 

8. Reports from T&E, exercises, and 
readiness reports 

4% 

n/a 

4% 

4% 

0% 

4% 

9. Combatant Commanders Input and 
Issues 

4% 

n/a 

4% 

4% 

4% 

4% 

c. Overarching DoD Docs. (Weight incl. 1- 
4) 

16% 

n/a 

16% 

16% 

16% 

16% 

1. Joint Doctrine 

4% 

n/a 

4% 

4% 

4% 

4% 

2. UJTL, Service task lists, METL 

4% 

n/a 

4% 

4% 

4% 

4% 

3. JOC 

4% 

n/a 

4% 

4% 

4% 

4% 

4. JFC 

4% 

n/a 

4% 

4% 

4% 

4% 

Total 

100% 

0% 

92% 

72% 

88% 

92% 


Table 26. Percent of Traceable Measures Summary Table 


PTM Evaluation Measure detailed results by alternative. 
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APPENDIX H. GLOSSARY OF TERMS 


Term 

Definition 

Action 

Something done (a task or activity.) 

Activity 

A function, mission, action, or collection of actions. 

Architecture 

A framework or structure that portrays relationships among all the 
elements of the subject force, system, or activity. 

Bandwidth 

The difference between the limiting frequencies of a continuous 
frequency band expressed in hertz (cycles per second). The term 
bandwidth is also loosely used to refer to the rate at which data can 
be transmitted over a given communications circuit. In the latter 
usage, bandwidth is usually expressed in either kilobits per second or 
megabits per second. 

C4ISR 

Command, control, communications, computers, intelligence, 
surveillance, and reconnaissance 

Capability 

The ability to execute a specified course of action. (A capability may 
or may not be accompanied by an intention.) 

Certification 

Certification is the process of confirming that a system or component 
complies with its specified requirements and is acceptable for 
operational use. 

Configuration 

Management 

A discipline applying technical and administrative direction and 
surveillance to: (1) identify and document the functional and physical 
characteristics of a configuration item; (2) control changes to those 
characteristics; and (3) record and report changes to processing and 
implementation status. 

Elasticity 

A measure of the % change in one variable (x - the numerator) with 
respect to another variable (y - the denominator.) 

% change x > % change in y = Elastic; 

% change in x < % change in y = Inelastic. 

Evaluation 

Measure 

Scale to measure the degree that we attain an objective 

Goal 

Threshold of achievement 

Interoperability 

The ability of systems, units, or forces to provide data, information, 
materiel, and services to and accept the same from other systems, 
units, or forces, and to use the data, information, materiel, and 
services so exchanged to enable them to operate effectively together. 

Measure of 
Effectiveness 

A criterion used to assess changes in system behavior, capability, or 
operational environment that is tied to measuring the attainment of an 
end state, achievement of an objective, or creation of an effect. 


209 




Term 

Definition 

Measures of 
Performance 

A criterion used to assess friendly actions that are tied to measuring 
task accomplishment. 

Network 

A group of components connected by communications. 

Objective 

Preferred direction of attainment of an evaluation consideration 

Policy 

A guiding principle or constraint, used to define general goals and 
acceptable criteria. 

Process 

Designated series of actions that produces some outcome. 

Project 

Management 

Plan 

A document that describes the background, goals, processes, 
resources, and schedule used to perform the project's tasks. 

Quality 

Assurance 

The process for monitoring a project to ensure that defined standards 
of quality are met. 

Reliability 

The probability that an item will perform its function under stated 
conditions of use and maintenance for a stated measure of the variant 
(time, distance, etc.) 

Requirement 

An approved need to achieve a capability, which is used in turn to 
accomplish tasks. 

Stakeholder 

An enterprise, organization, or individual having an interest or a stake 
in the outcome of the engineering of a system. 

System 

A combination of two or more interrelated pieces of equipment (or 
sets) arranged in a functional package to perform an operational 
function or to satisfy a requirement. 

Systems 

Architecture 

The organizational structure of a system or component, their 
relationships, and the principles and guidelines governing their design 
and evolution over time. 

Systems 

Engineering 

An interdisciplinary approach and means to enable the realization of 
successful systems. It focuses on defining customer needs and 
required functionality early in the development cycle, documenting 
requirements, then proceeding with design synthesis and system 
validation while considering the complete problem 

System 

Function 

An activity that the system must be designed to perform (Destroy 
targets) and an evaluation consideration for alternative system 
designs 
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Term 

Definition 

Validation 

The process of determining the degree to which a model or 
simulation is an accurate representation of the real world from the 
perspective of the intended uses of the model or simulation. 

Verification 

The process of determining that a model or simulation 
implementation accurately represents the developer’s conceptual 
description and specifications. 
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